ci: verify Woodpecker OAuth reconnection

Root cause: DNS returns IPv6 (AAAA) records for registry.npmjs.org
first. Alpine musl tries IPv6, which hangs because the cluster has
no IPv6 routing. NODE_OPTIONS=--dns-result-order=ipv4first forces
Node/npm to use IPv4 addresses.

.kiro/specs/intelligence-pipeline-v3/.config.kiro

@@ -0,0 +1 @@
+{"specId": "ce34e647-8d91-4295-a3c0-7b001abccdee", "workflowType": "requirements-first", "specType": "feature"}

.kiro/specs/intelligence-pipeline-v3/architecture-review.md

@@ -0,0 +1,69 @@
+# Stonks Oracle Intelligence Architecture Review
+
+## Recommendation
+
+Add generic OpenAI-compatible support, but implement it as a protocol/capability layer rather than a third vendor-specific branch. Keep Ollama native support. Convert the existing vLLM path into an OpenAI-compatible endpoint profile.
+
+For the RTX 4070 Ti SUPER cluster, do not replace the current 9B model with one smaller all-purpose model. Retain the 9B model as a focused adjudicator and split routine work into CPU-first specialist stages:
+
+1. Deterministic parsing and symbol-registry resolution.
+2. GLiNER2 Large for entities, event classes, relations, and evidence spans.
+3. FinBERT for company-specific financial sentiment probabilities.
+4. Retrieval-based novelty and duplicate detection.
+5. Calibrated confidence from observed field correctness.
+6. A stock-specific tabular model trained on realized abnormal returns for impact and horizon.
+7. The existing 9B Qwen-class model for ambiguous, causal, multi-company, or implied reasoning.
+
+This preserves the current reasoning ceiling, reduces average GPU inference, improves evidence fidelity, and adds stock-specific intelligence that a general language model cannot obtain from article text alone.
+
+## Option Review
+
+| Option | Best use | Weakness | Production role |
+|---|---|---|---|
+| Current Qwen3.5-class 9B monolith | Broad zero-shot semantics and hard reasoning | Expensive per document; stochastic; self-scores confidence/novelty/impact; weak calibration | Keep as adjudicator, not universal extractor |
+| Qwen3.5 4B | Smaller generalist | Lower reasoning ceiling with same architectural weaknesses | Benchmark only; not preferred |
+| NuExtract 1.5 3.8B | Literal schema filling | Limited implicit market reasoning; adds another generative runtime | Optional benchmark/fallback |
+| NuExtract 1.5 Smol 1.7B | Compact long-form extraction | Still autoregressive and not a sentiment/impact model | Optional CPU/on-demand filing stage |
+| NuExtract Tiny 0.5B | Very small extraction experiments | Accuracy ceiling too low for authoritative trading inputs without task tuning | Research/fine-tuning baseline |
+| GLiNER2 Large 340M | CPU-first entities, classes, relations, spans | Needs calibration and task-specific tuning for best results | Primary fast-path specialist |
+| FinBERT | Financial positive/negative/neutral probabilities | Not an extractor or reasoner | Per-company evidence sentiment |
+| Hybrid specialist + 9B | Routine precision plus retained hard-case intelligence | More engineering and observability work | Recommended architecture |
+| Hybrid + historical impact model | Text intelligence plus actual market-response learning | Requires leakage-safe dataset and monitoring | Best end-state |
+
+## Highest-Priority Existing Problems
+
+1. `services/extractor/vllm_client.py` ignores the supplied schema and requests only a generic JSON object.
+2. The vLLM default extraction temperature is `0.7`.
+3. Unknown provider values silently route to Ollama.
+4. Documents are truncated to 8,000 characters.
+5. The model is asked to invent authoritative novelty, confidence, impact, and horizon values.
+6. Those self-scores directly affect aggregation weighting.
+7. Provider attribution is hardcoded to Ollama.
+8. The extractor processes one job at a time at the application layer.
+9. Endpoint/model defaults conflict across code, migrations, Helm, and the vLLM deployment.
+10. A tracked Helm override contains plaintext production-like credentials and requires immediate rotation.
+
+## Expected Performance Shape
+
+The following are design targets to validate, not promises:
+
+- 60-80% of representative documents accepted through the CPU fast path after calibration.
+- 2x or greater reduction in GPU-seconds per accepted document.
+- Peak GPU memory near the current 9B deployment because no second generative model is permanently GPU-resident.
+- p50 latency substantially lower for routine documents.
+- p95 latency near the current model path for adjudicated documents.
+- Better exact-field and evidence accuracy from deterministic/specialist stages.
+- Same broad semantic ceiling because the 9B model remains available.
+- Better impact/horizon calibration once the historical outcome model is approved.
+
+## Immediate Next Decision
+
+The first implementation milestone should not be GLiNER integration. It should be:
+
+1. Rotate exposed credentials.
+2. Establish the real runtime model/configuration.
+3. Fix strict JSON Schema output and temperature on the current 9B endpoint.
+4. Build the gold corpus and replay harness.
+5. Then implement the gateway and specialist shadow path.
+
+That order creates a fair baseline and prevents the project from attributing simple request fixes to the new architecture.

.kiro/specs/intelligence-pipeline-v3/design.md

@@ -0,0 +1,792 @@
+# Design Document
+
+## Overview
+
+Intelligence Pipeline v3 replaces a monolithic "article to final trading-oriented JSON" request with a staged evidence and prediction architecture. The existing 9B model remains available, but its role changes from universal extractor and self-scorer to **semantic adjudicator** for the minority of documents that need broad language understanding.
+
+The design intentionally chooses the best long-term architecture rather than the minimum code change:
+
+- Generic OpenAI-compatible support is implemented as a capability-aware gateway, not another provider branch.
+- Explicit facts, entities, numbers, and sentiment are produced by CPU-first specialist components.
+- Novelty comes from retrieval and similarity.
+- Confidence comes from empirical calibration.
+- Impact and horizon come from a stock-specific model trained against realized outcomes.
+- The existing 9B vLLM model handles ambiguity, causality, implication, and conflicts.
+- Every field retains source evidence and model lineage.
+
+## Repository Review Findings
+
+The following findings materially shaped this design:
+
+| Finding | Repository location | Consequence |
+|---|---|---|
+| The vLLM client receives a JSON Schema but sends only `response_format: {"type": "json_object"}`. | `services/extractor/vllm_client.py:63-91` | The server is not constraining generation to the actual schema. |
+| vLLM extraction defaults to temperature `0.7`. | `services/shared/config.py:64-71`, `services/shared/config.py:284-291` | Routine extraction is needlessly stochastic. |
+| Unknown provider values silently fall back to Ollama. | `services/extractor/llm_factory.py:1-6`, `services/extractor/llm_factory.py:47-67` | Configuration mistakes can invoke the wrong endpoint without failing. |
+| Long documents are truncated to the first 8,000 characters. | `services/extractor/prompts.py:102-105` | Filings, transcripts, and long articles can lose material facts. |
+| The prompt asks one model for summary, entities, relevance, sentiment, impact, horizon, novelty, confidence, and evidence. | `services/extractor/prompts.py:107-126` | Extraction, reasoning, prediction, and self-evaluation are coupled. |
+| The prompt supplies tracked tickers and invites inferred sector/theme exposure. | `services/extractor/prompts.py:85-98` | Explicit mentions and inferred exposure are mixed before evidence validation. |
+| Persisted provider attribution is hardcoded to `ollama`, including failures. | `services/extractor/worker.py:166-184`, `services/extractor/worker.py:227-244` | Audit and model-performance attribution are incorrect for vLLM. |
+| A single worker loop pops and processes one job at a time. | `services/extractor/main.py:438-468`, invocation near `services/extractor/main.py:628` | Application-level parallelism is constrained even if vLLM supports batching. |
+| Runtime refresh mutates a client's private `_config`. | `services/extractor/main.py:496-531` | The protocol does not expose lifecycle or reconfiguration cleanly. |
+| The thesis rewriter reimplements Ollama/vLLM branching. | `services/recommendation/thesis_llm.py:87-200` | Provider support is duplicated and will continue drifting. |
+| Model defaults conflict across Python config, database migrations, Helm values, and the standalone vLLM deployment. | `services/shared/config.py`, `infra/migrations`, `infra/helm/stonks-oracle/values.yaml`, `infra/kube-vllm/deployment.yaml` | The repository cannot prove which model is canonical at runtime. |
+| Model-produced novelty and confidence directly affect aggregation weight; model-produced impact is reused as sentiment strength and impact. | `services/aggregation/scoring.py:436-529`, `services/aggregation/worker.py:430-463` | Uncalibrated model self-scores can materially influence downstream signals. |
+| A tracked Helm override contains plaintext production-like credentials. | `infra/helm/stonks-oracle/values-live-math.yaml` | Immediate rotation and history remediation are required before feature work ships. |
+
+The existing test suite around the LLM clients is useful. The focused provider tests passed after installing the declared dependencies plus the missing property-test dependency, but they encode current behavior and do not test true schema-constrained vLLM output.
+
+## Decision Summary
+
+### 1. Add generic OpenAI-compatible support
+
+Yes, but do not add an `OpenAIClient` beside `VLLMClient` and `OllamaClient`. Rename the concept:
+
+- `OllamaNativeClient` for `/api/chat` and Ollama-specific controls.
+- `OpenAICompatibleClient` for `/v1/chat/completions` and optionally `/v1/responses` after a separate compatibility gate.
+- `SpecialistHttpClient` for typed non-generative endpoints.
+
+`vllm` becomes a profile alias whose protocol is `openai_chat`. Hosted OpenAI, LM Studio, SGLang, LocalAI, or another compatible server can be represented by endpoint capabilities rather than new `if provider == ...` branches.
+
+Use direct `httpx` requests in the generic layer. This keeps the wire payload explicit, permits provider-specific `extra_body`, simplifies redacted request auditing, and avoids binding every compatible server to one SDK's assumptions.
+
+### 2. Retain the 9B model, but stop using it for every stage
+
+The RTX 4070 Ti SUPER remains dedicated to one 9B-class vLLM deployment. This preserves the current semantic ceiling and current peak VRAM class. The 9B model is invoked only for ambiguous cases and receives a compressed evidence packet rather than the raw entire document and ticker universe.
+
+### 3. Use a CPU-first fast path
+
+Run the following on CPU nodes:
+
+- GLiNER2 Large candidate for entities, event classes, relations, and schema-oriented extraction.
+- FinBERT candidate for company-specific positive/negative/neutral probabilities.
+- Deterministic parsers and the existing symbol registry for numeric facts and ticker identity.
+- Compact embeddings plus fingerprints for novelty and deduplication.
+- A calibrated tabular impact model for market direction, magnitude, and horizon.
+
+NuExtract 1.5 Smol is retained as an evaluated optional stage for long-form or hierarchical extraction. It is not made a second always-resident GPU model because the intended deployment should preserve the 9B model's GPU footprint and because GLiNER2 plus deterministic parsing may already cover most literal extraction.
+
+## Architecture
+
+```mermaid
+flowchart TD
+    A[Normalized document] --> B[Segmenter and offset map]
+    B --> C[Deterministic candidates\ncompany aliases, tickers, numbers, dates]
+    B --> D[GLiNER2 specialist\nentities, events, relations, facts]
+    C --> E[Symbol resolver]
+    D --> E
+    E --> F[Evidence linker and verifier]
+    F --> G[FinBERT per-company sentiment]
+    F --> H[Novelty and dedup retrieval]
+    G --> I[Confidence calibrator]
+    H --> I
+    I --> J{Fast-path acceptance?}
+    J -->|yes| K[Approved evidence graph]
+    J -->|no| L[9B Qwen adjudicator on vLLM]
+    L --> M[Post-adjudication verifier]
+    M --> K
+    K --> N[Stock-specific impact model]
+    N --> O[v3 intelligence records]
+    O --> P[Compatibility adapter]
+    P --> Q[Current aggregation and recommendation consumers]
+```
+
+### Why this can be more intelligent without a larger footprint
+
+A monolithic 9B model is broadly intelligent but is not necessarily the best estimator for every subproblem. The v3 design keeps that model for tasks requiring broad semantics while giving narrower jobs to components whose output can be calibrated and verified. The impact model adds information the language model does not have: observed historical market response. The result is not merely a smaller extractor; it is a system that combines textual reasoning with market-specific learned behavior.
+
+## Component Design
+
+### A. Inference Gateway
+
+#### Package layout
+
+```text
+services/shared/inference/
+├── protocol.py
+├── models.py
+├── registry.py
+├── router.py
+├── capabilities.py
+├── errors.py
+├── redaction.py
+└── clients/
+    ├── ollama_native.py
+    ├── openai_compatible.py
+    └── specialist_http.py
+```
+
+#### Core types
+
+```python
+@dataclass(frozen=True)
+class ProviderCapabilities:
+    chat_completions: bool
+    responses_api: bool
+    json_schema: bool
+    json_object: bool
+    seed: bool
+    usage: bool
+    max_completion_tokens: bool
+    reasoning_toggle: bool
+    model_listing: bool
+
+@dataclass(frozen=True)
+class InferenceTarget:
+    endpoint_id: UUID
+    deployment_id: UUID
+    protocol: Literal["ollama_native", "openai_chat", "specialist_http"]
+    base_url: str
+    model: str
+    capabilities: ProviderCapabilities
+    auth_secret_ref: str | None
+    extra_headers: Mapping[str, str]
+    extra_body: Mapping[str, Any]
+
+@dataclass
+class StructuredGenerationRequest:
+    messages: list[ChatMessage]
+    json_schema: dict[str, Any] | None
+    max_output_tokens: int
+    temperature: float = 0.0
+    seed: int | None = 0
+    timeout_seconds: float = 120.0
+    trace_id: str = ""
+
+@dataclass
+class InferenceResult:
+    content: str
+    parsed: dict[str, Any] | None
+    target: InferenceTarget
+    structured_mode: Literal["json_schema", "json_object", "prompt_only", "none"]
+    latency_ms: int
+    input_tokens: int | None
+    output_tokens: int | None
+    request_id: str | None
+    repaired: bool
+    retries: int
+```
+
+#### OpenAI-compatible structured output
+
+The client chooses the strongest declared mode:
+
+1. `json_schema`: send the actual schema and strict mode.
+2. `json_object`: allow only if the deployment profile explicitly permits it.
+3. `prompt_only`: allow only for experiments or legacy fallback.
+
+For current vLLM versions, the gateway should support both standard `response_format` JSON Schema and a configurable vLLM `structured_outputs` extra body because deployed versions may differ. The endpoint profile records which wire form passed its capability probe.
+
+Example standard payload:
+
+```json
+{
+  "model": "AxionML/Qwen3.5-9B-NVFP4",
+  "messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}],
+  "temperature": 0,
+  "max_tokens": 1536,
+  "response_format": {
+    "type": "json_schema",
+    "json_schema": {
+      "name": "adjudication_response",
+      "strict": true,
+      "schema": {}
+    }
+  }
+}
+```
+
+The gateway validates the parsed response again locally. Wire constraints reduce malformed output; they do not replace semantic validation.
+
+### B. Endpoint Registry
+
+#### New tables
+
+```sql
+CREATE TABLE inference_endpoints (
+    id UUID PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    protocol TEXT NOT NULL CHECK (protocol IN ('ollama_native','openai_chat','specialist_http')),
+    base_url TEXT NOT NULL,
+    auth_secret_ref TEXT,
+    auth_scheme TEXT NOT NULL DEFAULT 'bearer',
+    default_headers JSONB NOT NULL DEFAULT '{}',
+    health_path TEXT,
+    enabled BOOLEAN NOT NULL DEFAULT TRUE,
+    revision INTEGER NOT NULL DEFAULT 1,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE TABLE model_deployments (
+    id UUID PRIMARY KEY,
+    endpoint_id UUID NOT NULL REFERENCES inference_endpoints(id),
+    served_model_name TEXT NOT NULL,
+    display_name TEXT NOT NULL,
+    capabilities JSONB NOT NULL,
+    context_window INTEGER,
+    max_output_tokens INTEGER,
+    quantization TEXT,
+    runtime_metadata JSONB NOT NULL DEFAULT '{}',
+    enabled BOOLEAN NOT NULL DEFAULT TRUE,
+    revision INTEGER NOT NULL DEFAULT 1,
+    UNIQUE(endpoint_id, served_model_name)
+);
+
+CREATE TABLE agent_stage_bindings (
+    id UUID PRIMARY KEY,
+    agent_id UUID NOT NULL REFERENCES ai_agents(id),
+    stage TEXT NOT NULL,
+    model_deployment_id UUID REFERENCES model_deployments(id),
+    route_order INTEGER NOT NULL DEFAULT 0,
+    routing_config JSONB NOT NULL DEFAULT '{}',
+    is_active BOOLEAN NOT NULL DEFAULT TRUE,
+    revision INTEGER NOT NULL DEFAULT 1,
+    UNIQUE(agent_id, stage, route_order)
+);
+```
+
+Authentication values are not stored in these tables. `auth_secret_ref` identifies a mounted secret or environment key understood by the deployment.
+
+### C. Document Segmenter
+
+The segmenter replaces the 8,000-character prefix truncation.
+
+#### Output
+
+```python
+class DocumentChunk(BaseModel):
+    chunk_id: str
+    document_id: UUID
+    document_type: str
+    section_path: list[str]
+    speaker: str | None
+    start_char: int
+    end_char: int
+    text: str
+    overlap_left: int
+    overlap_right: int
+    boilerplate_score: float
+```
+
+Suggested initial limits:
+
+| Document type | Target chunk size | Overlap | Notes |
+|---|---:|---:|---|
+| News / press release | 700-1,000 tokens | 100 tokens | Preserve paragraph boundaries. |
+| Filing | 900-1,300 tokens | 150 tokens | Preserve headings and item sections. |
+| Transcript | 700-1,000 tokens | 100 tokens | Preserve speaker turns. |
+| Macro event | 500-800 tokens | 80 tokens | Favor compact event context. |
+
+The final values are benchmark parameters, not hard-coded assumptions.
+
+### D. Candidate Extraction and Symbol Resolution
+
+Deterministic parsers generate high-precision candidates before specialist inference:
+
+- Ticker tokens and exchange-qualified symbols.
+- Currency and number expressions, including `million`, `billion`, ranges, percentages, basis points, and per-share amounts.
+- Calendar and fiscal periods.
+- Comparison cues such as `up`, `down`, `beat`, `miss`, `raised`, `cut`, `above`, and `below`.
+- Company aliases from the symbol registry.
+
+GLiNER2 receives focused schemas and returns spans for entities, event classes, relations, and structured facts. The resolver merges deterministic and specialist candidates using source offsets, aliases, and local context.
+
+Explicit mentions and inferred exposures are different edge types:
+
+```text
+Document --explicitly_mentions--> Company
+Event --directly_affects--> Company
+Event --inferred_exposure--> Company
+Company --competes_with--> Company
+Company --supplies--> Company
+```
+
+Only explicit and verified direct effects enter the primary company extraction. Inferred exposure continues through the existing interpolation/propagation architecture with separate confidence and provenance.
+
+### E. Sentiment Stage
+
+FinBERT is run on company-linked evidence groups rather than the entire article. Each record contains:
+
+```python
+class CompanySentiment(BaseModel):
+    company_id: UUID
+    evidence_ids: list[UUID]
+    positive_probability: float
+    negative_probability: float
+    neutral_probability: float
+    calibration_version: str
+    model_deployment_id: UUID
+```
+
+Mixed sentiment is computed from multiple evidence groups and disagreement. It is not an unconstrained fourth softmax label.
+
+### F. Novelty Stage
+
+Novelty becomes retrieval-based:
+
+1. Compute an exact/near-duplicate fingerprint of normalized content.
+2. Embed document chunks and canonical company-event representations.
+3. Search a recent window in the vector index.
+4. Calculate document novelty and event novelty from nearest-neighbor similarity, duplicate count, source timing, and event identity.
+5. Store nearest matches for explainability.
+
+A compact embedding model will be selected in the evaluation harness. The implementation must keep the embedding backend replaceable and must not entangle novelty scoring with the generative endpoint.
+
+### G. Confidence and Routing
+
+#### Confidence features
+
+- Entity span score.
+- Alias-resolution margin between first and second candidate.
+- Numeric parser validity.
+- Evidence coverage.
+- Relation score.
+- Sentiment calibration confidence.
+- Cross-stage agreement.
+- Duplicate/novelty certainty.
+- Document completeness.
+- Document type and known hard-case patterns.
+
+A calibration artifact maps these features to field-level correctness probabilities. Routing uses both calibrated confidence and hard rules.
+
+#### Example adjudication triggers
+
+```text
+UNRESOLVED_ALIAS
+MULTIPLE_PRIMARY_COMPANIES
+CONTRADICTORY_NUMERIC_FACTS
+CONFLICTING_SENTIMENT
+IMPLIED_CAUSAL_IMPACT
+GUIDANCE_VS_CONSENSUS_REQUIRES_REASONING
+MATERIAL_FIELD_MISSING
+EVIDENCE_COVERAGE_BELOW_THRESHOLD
+CALIBRATED_CONFIDENCE_BELOW_THRESHOLD
+LONG_DOCUMENT_CROSS_CHUNK_RELATION
+```
+
+The desired initial target is 60-80 percent Fast_Path coverage after calibration. This is an evaluation target, not an assumed result.
+
+### H. Adjudication Packet
+
+The 9B model receives only the information necessary to resolve a specific ambiguity:
+
+```python
+class AdjudicationPacket(BaseModel):
+    document_id: UUID
+    document_type: str
+    question_codes: list[str]
+    candidate_companies: list[CompanyCandidate]
+    candidate_events: list[EventCandidate]
+    candidate_facts: list[FactCandidate]
+    candidate_sentiments: list[CompanySentiment]
+    evidence_spans: list[EvidenceSpan]
+    relevant_chunks: list[DocumentChunk]
+    required_decisions: list[str]
+```
+
+The adjudicator output does not contain final novelty, confidence, impact, or horizon. It resolves candidate identity, relationship, event interpretation, and supported qualitative direction. All output is evidence-linked and revalidated.
+
+### I. Impact and Horizon Model
+
+This is the highest-impact stock-specific change.
+
+#### Problem decomposition
+
+- **Text sentiment**: What tone or directional implication is supported by the document?
+- **Event identity**: What happened?
+- **Market impact**: How has this kind of event historically affected this kind of security in this market regime?
+- **Horizon**: Over what time window did the response usually manifest or decay?
+
+The current model conflates these. v3 separates them.
+
+#### Features
+
+- Event class probability vector.
+- Company-specific sentiment probability vector.
+- Numeric magnitude and normalized surprise when consensus or prior value exists.
+- Source credibility and historical source accuracy.
+- Novelty and duplicate count.
+- Evidence coverage and extraction uncertainty.
+- Document type.
+- Company sector, industry, market-cap bucket, liquidity, and beta.
+- Pre-event volatility, volume regime, and broad market regime.
+- Whether the event is direct, second-order, confirmed, quoted, or speculative.
+
+#### Labels
+
+Generate leakage-safe targets at defined event timestamps:
+
+- Signed abnormal return relative to an approved benchmark.
+- Absolute abnormal move.
+- Abnormal volume.
+- Direction labels for intraday, 1d, 7d, 30d, and 90d windows.
+- Time to peak response and decay where data quality supports it.
+
+#### Model family
+
+Start with a deterministic event-weight baseline plus a CPU tabular learner such as gradient-boosted trees. Calibrate class probabilities out-of-time. The model artifact and feature pipeline are versioned independently.
+
+The compatibility adapter may initially map expected signed magnitude to current `impact_score` and the most probable horizon to current `impact_horizon`, but richer distributions remain available to new consumers.
+
+### J. V3 Storage Schema
+
+Suggested logical records:
+
+```python
+class EvidenceSpan(BaseModel):
+    id: UUID
+    document_id: UUID
+    chunk_id: str
+    start_char: int
+    end_char: int
+    text: str
+    checksum: str
+
+class ExtractedEntity(BaseModel):
+    id: UUID
+    entity_type: str
+    literal_text: str
+    canonical_id: UUID | None
+    evidence_id: UUID
+    confidence: float
+    derivation: str
+
+class ExtractedFact(BaseModel):
+    id: UUID
+    fact_type: str
+    subject_entity_id: UUID | None
+    predicate: str
+    literal_value: str
+    normalized_value: dict | None
+    period: dict | None
+    evidence_ids: list[UUID]
+    confidence: float
+    derivation: str
+
+class CompanySignalCandidate(BaseModel):
+    company_id: UUID
+    relevance_probability: float
+    event_probabilities: dict[str, float]
+    sentiment_probabilities: dict[str, float]
+    direction_probabilities: dict[str, float]
+    horizon_probabilities: dict[str, float]
+    expected_magnitude: float | None
+    evidence_ids: list[UUID]
+    routing_reasons: list[str]
+    adjudicated: bool
+
+class StageLineage(BaseModel):
+    stage: str
+    endpoint_id: UUID | None
+    deployment_id: UUID | None
+    model_version: str | None
+    schema_version: str
+    calibration_version: str | None
+    started_at: datetime
+    duration_ms: int
+    status: str
+```
+
+### K. Compatibility Adapter
+
+The adapter creates current records without discarding v3 provenance:
+
+| Current field | V3 source |
+|---|---|
+| `summary` | Deterministic template or optional 9B narrative generated from approved facts. |
+| `macro_themes` | Approved event/theme classes. |
+| `novelty_score` | Retrieval-derived event/document novelty. |
+| `confidence` | Calibrated record correctness probability. |
+| `ticker` | Canonical symbol registry resolution. |
+| `relevance` | Calibrated direct-relevance probability. |
+| `sentiment` | Company-specific calibrated distribution mapped to legacy enum. |
+| `impact_score` | Approved impact-model magnitude mapped to legacy range. |
+| `impact_horizon` | Most probable approved horizon. |
+| `catalyst_type` | Versioned event taxonomy mapping. |
+| `evidence_spans` | Exact source spans. |
+
+The adapter marks `model_provider = 'hybrid'` and stores complete stage lineage separately. No provider identity is hardcoded.
+
+## Deployment Design for RTX 4070 Ti SUPER Cluster
+
+### GPU deployment
+
+One vLLM pod remains on the RTX 4070 Ti SUPER:
+
+```yaml
+resources:
+  limits:
+    nvidia.com/gpu: 1
+nodeSelector:
+  accelerator: rtx-4070-ti-super
+args:
+  - --model
+  - AxionML/Qwen3.5-9B-NVFP4
+  - --served-model-name
+  - stonks-adjudicator-9b
+  - --max-model-len
+  - "8192"
+  - --max-num-seqs
+  - "8"
+  - --gpu-memory-utilization
+  - "0.80"
+  - --structured-outputs-config.backend
+  - auto
+```
+
+Exact flags must match the pinned vLLM version. The deployment test must verify strict schema output before promotion.
+
+### CPU specialist deployment
+
+```yaml
+replicas: 2
+resources:
+  requests:
+    cpu: "2"
+    memory: 4Gi
+  limits:
+    cpu: "6"
+    memory: 10Gi
+```
+
+Initial pod contents:
+
+- GLiNER2 Large.
+- FinBERT.
+- Tokenizers and deterministic parsers.
+- Optional embedding model.
+
+NuExtract 1.5 Smol should run as a separate benchmark or on-demand CPU deployment so its value can be measured independently.
+
+### Queue topology
+
+```text
+extraction.incoming
+    -> intelligence.router
+        -> extraction.fast
+        -> extraction.adjudication
+        -> extraction.persist
+        -> extraction.review
+```
+
+The router owns document state transitions. Workers use leases and idempotency keys so a retry cannot create duplicate intelligence records.
+
+### Concurrency
+
+- Fast path: configurable worker pool, initially 4-8 concurrent documents per pod.
+- Specialist API: micro-batching bounded by maximum wait time.
+- Adjudicator: application semaphore aligned with vLLM `max-num-seqs` and measured KV-cache behavior.
+- Persistence: independent bounded pool.
+
+## OpenAI-Compatible Support Details
+
+### Profiles
+
+| Profile | Protocol | Typical use |
+|---|---|---|
+| `ollama` | `ollama_native` | Existing Ollama endpoint. |
+| `vllm` | `openai_chat` | Backward-compatible alias using vLLM capability profile. |
+| `openai` | `openai_chat` | Hosted OpenAI endpoint with secret reference and egress policy. |
+| `openai_compatible` | `openai_chat` | Any explicitly configured compatible server. |
+| `specialist` | `specialist_http` | Typed GLiNER/FinBERT service. |
+
+### Capability probes
+
+A deployment activation test performs:
+
+1. Health request.
+2. Model listing if supported.
+3. Minimal chat request.
+4. Strict JSON Schema request.
+5. Usage metadata check.
+6. Seed/determinism check if declared.
+7. Maximum output field compatibility check.
+
+Probe results are stored with timestamp and software version. A failed capability cannot be enabled merely by selecting it in the UI.
+
+### Egress and data policy
+
+Endpoint profiles include data-handling classification:
+
+```text
+local_private
+cluster_private
+approved_external
+forbidden_for_sensitive_docs
+```
+
+External endpoints are disabled by default. Routing to an approved external endpoint requires both an active binding and a document policy permitting egress.
+
+## Evaluation Strategy
+
+### Gold corpus
+
+Build a minimum initial corpus of 1,000 human-reviewed documents, stratified across:
+
+- News, filings, transcripts, and press releases.
+- Single-company and multi-company stories.
+- Earnings beats/misses and guidance changes.
+- M&A, legal, regulatory, product, supply-chain, rating, management, and macro events.
+- Explicit facts versus implied consequences.
+- Short and long documents.
+- Duplicate and recycled stories.
+
+### Compared systems
+
+1. Current production path with current model and current prompt.
+2. Current 9B model with corrected temperature and strict JSON Schema.
+3. GLiNER2 + deterministic extraction.
+4. GLiNER2 + deterministic extraction + FinBERT.
+5. Optional NuExtract 1.5 Smol extraction path.
+6. Full v3 fast path.
+7. Full v3 with 9B adjudication.
+
+This separation prevents architecture gains from being confused with a simple fix to the current vLLM request.
+
+### Metrics
+
+- Company/ticker precision, recall, F1.
+- Event macro-F1 and per-class F1.
+- Numeric exact match with tolerance-aware normalization.
+- Relation F1.
+- Evidence support and offset validity.
+- Company-specific sentiment macro-F1.
+- Confidence ECE and Brier score.
+- Unsupported-claim rate.
+- JSON/schema failure rate.
+- Fast-path coverage.
+- Adjudication reason distribution.
+- p50/p95 latency and documents per minute.
+- CPU-seconds, GPU-seconds, tokens, and peak GPU memory per document.
+- Impact-model direction accuracy, calibration, rank correlation, and out-of-time error by horizon.
+
+### Promotion sequence
+
+1. Correct current vLLM schema constraints and temperature; establish baseline.
+2. Run v3 offline replay.
+3. Run v3 shadow mode.
+4. Enable v3 for audit-only UI.
+5. Canary compatibility outputs for a small percentage of non-trading downstream traffic.
+6. Canary signal influence with automatic rollback.
+7. Promote by document type and confidence tier.
+8. Retire v2 only after a separately reviewed milestone.
+
+## Security Design
+
+### Immediate blocker
+
+The repository contains plaintext production-like credentials in a tracked Helm values file. Treat them as compromised:
+
+1. Rotate the database, object-store, Redis, broker, and market-data credentials.
+2. Disable or replace old keys.
+3. Remove secret values from tracked files.
+4. Purge historical values using an approved Git history rewrite process.
+5. Migrate to an external secret manager.
+6. Add secret scanning to local hooks and CI.
+7. Audit access logs for the affected credentials.
+
+No values are reproduced in this specification.
+
+### Inference security
+
+- Secrets are resolved only at runtime.
+- Request logging records hashes and metadata, not authorization values.
+- Raw source text is not logged at INFO level.
+- External endpoint use is policy-gated.
+- Stored raw prompts and responses use restricted object-store buckets and retention policies.
+- Provider errors are normalized to avoid echoing secret-bearing response headers.
+
+## Testing Strategy
+
+### Unit tests
+
+- Endpoint capability selection.
+- Strict schema payload construction.
+- Header and error redaction.
+- Segment offset round trips.
+- Numeric normalization.
+- Alias resolution and ambiguity margins.
+- Evidence linkage.
+- Compatibility mappings.
+- Confidence feature construction.
+- Routing rules.
+
+### Property-based tests
+
+- Every Evidence_Span round-trips to identical source text.
+- Normalization never changes the literal stored value.
+- Unknown providers always fail closed.
+- Credentials never appear in serialized errors or logs.
+- Compatibility mappings remain bounded in legacy field ranges.
+- Reprocessing the same document and model versions is idempotent.
+- Route decisions are deterministic for identical calibrated inputs.
+
+### Contract tests
+
+- Ollama native endpoint.
+- vLLM OpenAI-compatible endpoint.
+- Mock hosted OpenAI-compatible endpoint.
+- Specialist service schemas.
+- Capability probe behavior across supported structured-output modes.
+
+### Integration tests
+
+- Full document through fast path.
+- Full document through adjudication path.
+- Adjudicator outage with safe fast-path handling.
+- Long filing crossing multiple chunks.
+- Multi-company article with opposing sentiment.
+- Duplicate story and novelty calculation.
+- Rollback from v3 to v2.
+
+### Load tests
+
+- CPU specialist batching.
+- Queue backpressure.
+- vLLM concurrent adjudication.
+- Peak 4070 Ti SUPER memory.
+- End-to-end throughput at representative article arrival rates.
+
+## Migration Plan
+
+### Phase 0: Security and baseline
+
+Rotate secrets, add scanning, pin current runtime configuration, and benchmark the existing path.
+
+### Phase 1: Provider foundation
+
+Add the Inference Gateway and registry. Convert current extractor, classifier, and thesis rewriter. Keep behavior otherwise equivalent.
+
+### Phase 2: Correct current generative extraction
+
+Use strict schema-constrained vLLM output, temperature zero, accurate provider lineage, and bounded output sizes. This produces a fair current baseline.
+
+### Phase 3: V3 data and specialist shadow path
+
+Add segmentation, v3 storage, deterministic extraction, GLiNER2, FinBERT, novelty, confidence, and audit UI. Persist shadow outputs only.
+
+### Phase 4: Adjudication and impact model
+
+Add confidence routing, focused 9B adjudication, historical outcome features, deterministic impact baseline, and trained impact model.
+
+### Phase 5: Canary and promotion
+
+Enable compatibility outputs by percentage and document type, then gradually allow v3 signals into aggregation.
+
+### Phase 6: Fine-tuning and cleanup
+
+Fine-tune specialist models from reviewed cases, increase fast-path coverage, then remove deprecated v2 code in a separate change.
+
+## Risks and Mitigations
+
+| Risk | Mitigation |
+|---|---|
+| Specialist model misses implicit meaning. | Retain 9B adjudication with calibrated routing. |
+| Complexity creates more failure modes. | Typed stage contracts, idempotent queues, stage-level metrics, and safe fallback. |
+| Fast-path confidence is overestimated. | Held-out calibration, conservative thresholds, and shadow review. |
+| Impact model learns leakage or regime artifacts. | Event-time feature snapshots, out-of-time validation, per-regime monitoring, and immutable predictions. |
+| OpenAI-compatible servers differ subtly. | Capability probes and profile-specific wire settings, not optimistic assumptions. |
+| A second model increases memory. | Keep specialists on CPU and make NuExtract optional/on-demand. |
+| Existing downstream code assumes one model output. | Compatibility adapter and additive migrations. |
+| Reviewer labels become inconsistent. | Annotation guide, double review for hard cases, and inter-annotator agreement tracking. |

.kiro/specs/intelligence-pipeline-v3/requirements.md

@@ -0,0 +1,314 @@
+# Requirements Document
+
+## Introduction
+
+Stonks Oracle currently asks a general-purpose generative model to perform entity discovery, ticker attribution, fact extraction, event classification, sentiment analysis, novelty estimation, confidence estimation, impact scoring, horizon selection, evidence quoting, and summarization in one response. That design is convenient, but it couples factual extraction to generative sampling and allows uncalibrated model self-assessments to influence signal weighting.
+
+This specification introduces **Intelligence Pipeline v3**, a multi-stage, evidence-grounded inference system that preserves the current 9B model's reasoning ability for genuinely ambiguous documents while moving routine extraction, sentiment, novelty, confidence, and impact estimation into specialized and calibratable components. The target deployment retains the existing RTX 4070 Ti SUPER vLLM footprint and uses CPU-first specialist services for the fast path.
+
+The specification also replaces provider-specific branching with a capability-aware inference gateway supporting Ollama native endpoints and generic OpenAI-compatible endpoints, including vLLM and hosted OpenAI-compatible services.
+
+## Goals
+
+1. Match or exceed the current 9B pipeline's field-level accuracy and reasoning ceiling.
+2. Reduce average GPU work per document without increasing peak GPU memory materially.
+3. Make every extracted fact traceable to evidence in the source document.
+4. Replace model-generated confidence, novelty, impact, and horizon values with calibrated or deterministic values.
+5. Support generic OpenAI-compatible inference without adding another duplicated provider branch.
+6. Establish a measurable benchmark, shadow rollout, and promotion process.
+7. Preserve downstream compatibility while the v2 schema and database consumers are migrated.
+
+## Non-Goals
+
+1. Replacing the existing recommendation, risk, or trading engines in one release.
+2. Removing the current 9B model before the v3 pipeline passes promotion gates.
+3. Treating backtest profit alone as proof of extraction correctness.
+4. Sending credentials or proprietary documents to external providers by default.
+5. Requiring a second GPU-resident generative model.
+
+## Glossary
+
+- **Inference_Gateway**: Shared client and routing layer that invokes Ollama-native, OpenAI-compatible, and specialist inference endpoints through one typed interface.
+- **Endpoint_Profile**: Persisted endpoint configuration containing protocol, URL, authentication reference, capabilities, and health settings.
+- **Model_Deployment**: A model served by an Endpoint_Profile with declared capabilities and limits.
+- **Pipeline_Stage**: One step in Intelligence Pipeline v3, such as segmentation, extraction, sentiment, verification, novelty, adjudication, or impact prediction.
+- **Fast_Path**: CPU-first processing that completes without invoking the 9B generative model.
+- **Adjudication_Path**: Processing that invokes the 9B model because evidence is ambiguous, contradictory, incomplete, or semantically complex.
+- **Evidence_Span**: Exact source text plus stable character offsets and a chunk identifier.
+- **Candidate**: A proposed entity, fact, event, sentiment, or relation before validation and calibration.
+- **Calibrated_Confidence**: Probability-like confidence derived from validation data, not a number supplied by a generative model.
+- **Impact_Model**: A lightweight supervised model that estimates signed market impact and horizon from extracted features and historical outcomes.
+- **Compatibility_Adapter**: Mapper from v3 records to the current v2 `document_intelligence` and `document_impact_records` structures.
+- **Gold_Corpus**: Human-reviewed documents and field-level labels used for acceptance testing.
+- **Shadow_Mode**: Running v3 alongside the current pipeline without allowing v3 outputs to affect production decisions.
+
+## Requirements
+
+### Requirement 1: Secure Baseline and Credential Remediation
+
+**User Story:** As an operator, I want repository and deployment credentials handled through secret stores, so that model-pipeline improvements do not ship on top of exposed credentials.
+
+#### Acceptance Criteria
+
+1. THE Team SHALL rotate every credential currently stored as plaintext in tracked repository files before deploying Intelligence Pipeline v3.
+2. THE Repository SHALL remove plaintext database, object-store, Redis, broker, and market-data credentials from tracked Helm values and Git history.
+3. THE Deployment SHALL reference credentials through Kubernetes Secrets populated by External Secrets, SOPS, Sealed Secrets, or an equivalent approved mechanism.
+4. THE CI_Pipeline SHALL run secret scanning on pull requests and protected branches.
+5. IF secret scanning detects a high-confidence credential, THEN THE CI_Pipeline SHALL fail before packaging or deployment.
+6. THE Documentation SHALL record the rotation date and affected secret names without recording secret values.
+
+### Requirement 2: Capability-Aware Generic Inference Gateway
+
+**User Story:** As a developer, I want one inference abstraction that supports Ollama and generic OpenAI-compatible services, so that endpoints can be changed without duplicating business logic.
+
+#### Acceptance Criteria
+
+1. THE Inference_Gateway SHALL support the protocols `ollama_native`, `openai_chat`, and `specialist_http`.
+2. THE Inference_Gateway SHALL treat `vllm` as a backward-compatible profile alias for `openai_chat`, not as a separate client implementation.
+3. WHEN an `openai_chat` request requires structured output and the endpoint declares `json_schema` support, THE Inference_Gateway SHALL send the complete supplied JSON Schema in strict structured-output mode.
+4. WHEN an endpoint supports only JSON-object mode, THE Inference_Gateway SHALL use JSON-object mode only when that fallback is explicitly enabled for the Model_Deployment.
+5. WHEN neither schema nor JSON-object constraints are supported, THE Inference_Gateway SHALL use prompt-only JSON generation only when explicitly enabled and SHALL mark the response as unconstrained.
+6. IF a provider or protocol value is unknown, THEN THE Inference_Gateway SHALL fail closed with a configuration error and SHALL NOT silently route to Ollama.
+7. THE Inference_Gateway SHALL support configurable base URL, request path, API-key secret reference, authorization scheme, additional headers, timeouts, retries, concurrency limit, and provider-specific extra request fields.
+8. THE Inference_Gateway SHALL redact authentication values and configured sensitive headers from logs, traces, and stored request snapshots.
+9. THE Inference_Gateway SHALL expose typed response metadata including endpoint ID, deployment ID, model name, protocol, request ID, latency, token usage, structured-output mode, retry count, and error category.
+10. THE Inference_Gateway SHALL provide health and capability probes and cache their results with a bounded TTL.
+11. WHEN endpoint capabilities are changed or a probe fails, THE Router SHALL invalidate the cached capability record before the next invocation.
+12. THE Existing thesis rewriter, event classifier, and document extractor SHALL use the same Inference_Gateway rather than implementing separate Ollama/vLLM branches.
+
+### Requirement 3: Canonical Endpoint and Model Registry
+
+**User Story:** As an operator, I want the database and UI to identify exactly which endpoint and model serve each stage, so that environment, migration, Helm, and runtime defaults cannot drift silently.
+
+#### Acceptance Criteria
+
+1. THE Database SHALL store `inference_endpoints`, `model_deployments`, and `agent_stage_bindings` as canonical runtime records.
+2. EACH Inference_Endpoint SHALL include name, protocol, base URL, authentication secret reference, health path, default headers, enabled state, and timestamps.
+3. EACH Model_Deployment SHALL include endpoint ID, served model name, display name, capabilities, context limit, output limit, quantization, structured-output modes, and enabled state.
+4. EACH Agent_Stage_Binding SHALL map an agent and pipeline stage to one or more ordered Model_Deployments plus routing configuration.
+5. WHEN runtime configuration is resolved, THE Service SHALL record the exact endpoint, deployment, and binding revision used.
+6. THE API SHALL validate endpoint URLs, protocol values, capability declarations, and model names before activation.
+7. THE UI SHALL use controlled protocol and endpoint selections rather than an unrestricted provider text field.
+8. THE Migration SHALL translate existing `ollama` and `vllm` agent settings into Endpoint_Profile and Model_Deployment records without breaking active agents.
+9. THE Application SHALL have one documented fallback configuration source; conflicting model defaults in code, migrations, and Helm SHALL be removed.
+
+### Requirement 4: Document Segmentation and Source Preservation
+
+**User Story:** As an analyst, I want long articles, filings, and transcripts processed without destructive truncation, so that material facts near the end of a document are not lost.
+
+#### Acceptance Criteria
+
+1. THE Pipeline SHALL preserve the full normalized source document and SHALL NOT truncate it to a fixed character prefix for extraction.
+2. THE Segmenter SHALL create sentence-aware chunks with stable chunk IDs, source character offsets, and configurable overlap.
+3. THE Segmenter SHALL use document-type-specific chunk sizes for articles, filings, transcripts, and press releases.
+4. THE Segmenter SHALL preserve headings, speaker labels, table-derived text markers, and section boundaries when present.
+5. WHEN duplicate or boilerplate sections are detected, THE Segmenter SHALL mark them without deleting the only occurrence of a fact.
+6. THE Pipeline SHALL retain a mapping from every downstream Evidence_Span to the original document offsets.
+7. IF a document cannot be decoded or segmented, THEN THE Pipeline SHALL mark the document as a typed preprocessing failure and SHALL NOT fabricate an empty extraction.
+
+### Requirement 5: Deterministic Candidate Generation and Ticker Resolution
+
+**User Story:** As a signal consumer, I want explicit companies and numeric facts resolved deterministically where possible, so that a language model is not asked to invent identifiers or parse trivial values.
+
+#### Acceptance Criteria
+
+1. THE Candidate_Generator SHALL detect explicit ticker symbols, company names, aliases, executives, products, currencies, percentages, dates, ranges, EPS values, revenue values, guidance values, and common financial ratios.
+2. THE Symbol_Resolver SHALL use the existing company and symbol registry as the source of truth for ticker identity.
+3. THE Pipeline SHALL distinguish explicit company mentions from inferred exposure relationships.
+4. THE Pipeline SHALL NOT pass the entire tracked-ticker universe to a generative prompt.
+5. WHEN multiple companies match an alias, THE Symbol_Resolver SHALL return ranked candidates and SHALL require contextual disambiguation or adjudication.
+6. WHEN a ticker is not present in the symbol registry, THE Pipeline SHALL preserve the literal mention as unresolved rather than inventing a registered ticker.
+7. THE Numeric_Normalizer SHALL retain both literal source text and normalized values, currencies, units, periods, and ranges.
+8. THE Pipeline SHALL reject normalized numeric facts whose value cannot be traced to an Evidence_Span.
+
+### Requirement 6: Specialist Extraction Service
+
+**User Story:** As an operator, I want routine entity, event, relation, and fact extraction to run on CPU-first specialist models, so that GPU capacity is reserved for difficult reasoning.
+
+#### Acceptance Criteria
+
+1. THE Specialist_Service SHALL expose batched APIs for entity extraction, schema extraction, relation extraction, and text classification.
+2. THE Initial specialist extractor SHALL support company, person, product, event, financial metric, date, percentage, currency, and relationship schemas.
+3. THE Specialist_Service SHALL return character spans and per-candidate scores for every extracted item.
+4. THE Specialist_Service SHALL run without requiring the RTX 4070 Ti SUPER.
+5. THE Initial deployment SHALL evaluate GLiNER2 Large as the primary unified extraction and classification model.
+6. THE Benchmark SHALL evaluate NuExtract 1.5 Smol as an optional long-form or hierarchical fact-extraction stage, but it SHALL NOT become an always-resident GPU model without passing incremental-value and resource gates.
+7. THE Specialist_Service SHALL support model version pinning, warm-up, health checks, bounded batching, and graceful degradation.
+8. WHEN specialist inference fails, THE Router SHALL either retry according to policy or route to adjudication; it SHALL record the failure and SHALL NOT silently substitute default facts.
+9. THE Specialist_Service SHALL expose model and schema versions in every response.
+
+### Requirement 7: Company-Specific Financial Sentiment
+
+**User Story:** As an analyst, I want sentiment tied to each company and supporting evidence, so that a positive statement about one firm is not applied to every company in the article.
+
+#### Acceptance Criteria
+
+1. THE Sentiment_Stage SHALL score evidence sentences or evidence groups associated with each resolved company.
+2. THE Initial sentiment classifier SHALL evaluate FinBERT as the baseline financial-domain model.
+3. THE Sentiment_Stage SHALL return positive, negative, and neutral probabilities rather than only a discrete label.
+4. THE Pipeline SHALL derive mixed sentiment from conflicting supported evidence, not from an unconstrained model label.
+5. WHEN an article mentions competitors with opposing effects, THE Pipeline SHALL produce separate company-specific sentiment records.
+6. THE Sentiment_Stage SHALL preserve the evidence IDs used for each probability distribution.
+7. THE Production model SHALL be calibrated on the Gold_Corpus before its probabilities are treated as confidence values.
+
+### Requirement 8: Evidence Verification and Grounding
+
+**User Story:** As an auditor, I want every material claim verified against source evidence, so that generated summaries and signals cannot rely on unsupported assertions.
+
+#### Acceptance Criteria
+
+1. EVERY material company fact, event, amount, direction, and relationship SHALL reference one or more Evidence_Spans.
+2. THE Verifier SHALL check span validity, source offsets, entity association, and schema compatibility.
+3. THE Benchmark SHALL evaluate a compact entailment verifier for claims that require semantic validation beyond exact matching.
+4. IF a candidate conflicts with its evidence, THEN THE Pipeline SHALL reject it or route the conflict to adjudication.
+5. THE Pipeline SHALL calculate evidence coverage as the proportion of required fields supported by valid spans.
+6. THE Pipeline SHALL store rejected candidates and rejection reasons for audit and active learning.
+7. JSON repair SHALL NOT transform an unsupported or truncated generative answer into a valid production extraction without marking it as repaired and revalidating every material field.
+
+### Requirement 9: Deterministic Novelty and Duplicate Detection
+
+**User Story:** As a signal consumer, I want novelty based on comparison with recent information, so that a model's subjective novelty guess does not amplify repeated news.
+
+#### Acceptance Criteria
+
+1. THE Novelty_Stage SHALL compare each document and material event against a configurable recent-history window.
+2. THE Novelty_Stage SHALL combine exact/near-duplicate fingerprints with compact semantic embeddings.
+3. THE Pipeline SHALL calculate novelty separately for document-level content and company-event content.
+4. THE Novelty_Stage SHALL return nearest matching document or event IDs plus similarity scores.
+5. THE Pipeline SHALL derive `novelty_score` from the similarity distribution and duplicate count using a versioned deterministic formula or calibrated model.
+6. A generative model SHALL NOT provide the authoritative novelty value used by aggregation.
+7. WHEN novelty cannot be calculated because history is unavailable, THE Pipeline SHALL use a conservative versioned default and mark the reason.
+
+### Requirement 10: Calibrated Extraction Confidence
+
+**User Story:** As a downstream scorer, I want confidence to reflect observed correctness, so that the system does not trust a model merely because it reports confidence in itself.
+
+#### Acceptance Criteria
+
+1. THE Pipeline SHALL calculate field-level and record-level confidence from specialist scores, symbol resolution, evidence validation, schema completeness, model agreement, and historical calibration.
+2. A generative model's self-reported confidence SHALL NOT be used as authoritative extraction confidence.
+3. THE Calibration_Process SHALL evaluate isotonic, Platt, or equivalent calibration methods on held-out Gold_Corpus data.
+4. THE Pipeline SHALL report Expected Calibration Error and Brier score for probability-bearing stages.
+5. THE Router SHALL use calibrated uncertainty and explicit conflict rules to choose Fast_Path or Adjudication_Path.
+6. THE Pipeline SHALL retain stage-level confidence components for explainability.
+7. WHEN calibration data is insufficient for a class, THE Pipeline SHALL use conservative thresholds and mark the class as under-calibrated.
+
+### Requirement 11: 9B Generative Adjudicator
+
+**User Story:** As an analyst, I want the current reasoning capability retained for hard documents, so that specialization does not reduce intelligence on nuanced cases.
+
+#### Acceptance Criteria
+
+1. THE Adjudicator SHALL initially use the existing 9B-class model served by vLLM on the RTX 4070 Ti SUPER.
+2. THE Adjudicator SHALL receive selected source chunks, Evidence_Spans, candidate facts, candidate probabilities, conflicts, and a precise adjudication question rather than the entire tracked ticker list.
+3. THE Adjudicator SHALL use strict JSON Schema constrained output when supported by the endpoint.
+4. THE Adjudicator SHALL use deterministic generation settings appropriate for extraction, including a production default temperature of zero unless a benchmark proves a different value superior.
+5. THE Adjudicator SHALL NOT be asked to provide authoritative novelty, confidence, or impact values.
+6. THE Router SHALL invoke adjudication for unresolved entity aliases, contradictory evidence, multi-company causal relationships, implied consequences, complex guidance, materially incomplete fast-path results, or low calibrated confidence.
+7. THE Adjudicator SHALL return field-level decisions, evidence references, and decision reasons.
+8. IF adjudication output references evidence not supplied to it, THEN THE Verifier SHALL reject the unsupported field.
+9. THE Adjudicator SHALL remain optional for thesis prose; deterministic signal records SHALL not depend on prose generation succeeding.
+10. THE Peak GPU memory budget SHALL not exceed the measured current 9B deployment baseline by more than 5 percent unless explicitly approved.
+
+### Requirement 12: Stock-Specific Impact and Horizon Model
+
+**User Story:** As a trader, I want impact and horizon estimated from historical market behavior rather than language-model intuition, so that signals are tied to observed outcomes.
+
+#### Acceptance Criteria
+
+1. THE Pipeline SHALL separate textual sentiment from expected market impact.
+2. THE Impact_Model SHALL consume versioned features including event type probabilities, sentiment probabilities, magnitude, surprise where available, source history, novelty, company attributes, market regime, pre-event volatility, and evidence quality.
+3. THE Training_Pipeline SHALL create leakage-safe labels from abnormal returns and volume responses over configured horizons.
+4. THE Initial model family SHALL be a CPU-efficient calibrated tabular model and SHALL include a transparent deterministic baseline.
+5. THE Impact_Model SHALL output signed direction probabilities, expected magnitude, horizon probabilities, and model uncertainty.
+6. THE Production model SHALL be evaluated out-of-time and by event type, sector, market-cap bucket, and source.
+7. THE Pipeline SHALL preserve existing downstream fields through a Compatibility_Adapter while storing richer probability distributions in v3 tables.
+8. IF no trained Impact_Model is approved, THEN THE Pipeline SHALL use the deterministic baseline and SHALL NOT fall back to a generative model's impact score.
+9. THE Outcome_Evaluator SHALL feed realized outcomes back into model monitoring and retraining datasets without mutating historical predictions.
+10. THE Pipeline SHALL version feature definitions, training data ranges, model artifacts, thresholds, and calibration artifacts.
+
+### Requirement 13: Versioned Intelligence Schema and Provenance
+
+**User Story:** As a developer, I want a richer schema with field-level provenance, so that downstream consumers can distinguish facts, probabilities, decisions, and generated prose.
+
+#### Acceptance Criteria
+
+1. THE Database SHALL store v3 entities, facts, evidence spans, company signal candidates, stage runs, adjudication decisions, and model lineage in normalized or well-defined JSONB-backed tables.
+2. EVERY v3 field SHALL identify whether it is deterministic, specialist-derived, adjudicated, calibrated, or compatibility-derived.
+3. EVERY stage run SHALL record input references, output references, model versions, endpoint identity, duration, error state, and trace ID.
+4. THE Compatibility_Adapter SHALL map approved v3 outputs to existing v2 persistence records during migration.
+5. THE Compatibility_Adapter SHALL identify its own version and SHALL not overwrite original v3 probabilities.
+6. THE persisted `model_provider` and model lineage SHALL reflect the actual route used and SHALL not be hardcoded to Ollama.
+7. THE Pipeline SHALL retain raw model output only in approved object storage with configured retention and access controls.
+
+### Requirement 14: Parallelism, Queues, and Resource Isolation
+
+**User Story:** As an operator, I want parallel throughput without saturating the GPU or blocking unrelated stages, so that the cluster remains responsive.
+
+#### Acceptance Criteria
+
+1. THE Extractor SHALL support multiple in-flight documents using bounded asynchronous workers rather than a single unbounded sequential loop.
+2. THE Fast_Path and Adjudication_Path SHALL have separate queue or concurrency controls.
+3. THE Specialist_Service SHALL support dynamic batching within configured latency limits.
+4. THE Adjudicator SHALL enforce a GPU-safe concurrency semaphore coordinated with vLLM limits.
+5. THE Router SHALL apply backpressure when either path exceeds its queue-depth or latency thresholds.
+6. THE Deployment SHALL assign specialist workloads to CPU nodes and the 9B vLLM workload to the RTX 4070 Ti SUPER node by default.
+7. THE System SHALL expose queue depth, service time, batch size, GPU memory, GPU utilization, fast-path rate, and adjudication rate.
+8. WHEN the adjudicator is unavailable, THE Pipeline SHALL continue only for documents meeting a conservative fast-path acceptance threshold; all others SHALL remain queued or fail safely.
+
+### Requirement 15: Observability, Audit, and Explainability
+
+**User Story:** As an operator and analyst, I want to understand why a document produced a signal and which component made each decision.
+
+#### Acceptance Criteria
+
+1. THE Pipeline SHALL emit one distributed trace covering preprocessing, specialist stages, routing, adjudication, impact prediction, and persistence.
+2. THE Metrics SHALL include field validity, evidence coverage, entity resolution rate, sentiment agreement, calibration metrics, fast-path coverage, adjudication causes, schema failure rate, latency percentiles, token usage, and GPU-seconds per document.
+3. THE Audit API SHALL return model lineage and evidence for a document, company, and generated signal.
+4. THE UI SHALL distinguish observed facts, inferred exposure, sentiment, predicted impact, and generated narrative.
+5. THE Pipeline SHALL store routing reasons as structured codes rather than log-only text.
+6. THE Pipeline SHALL allow a reviewer to mark a field correct, incorrect, unsupported, or ambiguous and add a corrected value.
+7. Reviewer corrections SHALL be immutable audit events and SHALL feed the active-learning dataset only through an approved export process.
+
+### Requirement 16: Benchmark, Shadow Mode, and Promotion Gates
+
+**User Story:** As an owner, I want the new architecture proven against the current system before it affects trades, so that complexity is justified by measured improvement.
+
+#### Acceptance Criteria
+
+1. THE Team SHALL create a versioned Gold_Corpus covering articles, filings, press releases, transcripts, macro news, multi-company stories, contradictory reports, and long documents.
+2. THE Evaluation_Harness SHALL run the current pipeline and every proposed v3 configuration on identical inputs.
+3. THE Evaluation SHALL report field precision, recall, F1, exact-match accuracy, evidence support rate, ticker-resolution accuracy, event macro-F1, sentiment macro-F1, calibration, latency, throughput, CPU use, GPU use, and cost.
+4. THE Evaluation SHALL report results by document type, event class, source, sector, and difficulty bucket.
+5. THE Initial promotion gate SHALL require no statistically meaningful regression in any safety-critical field and measurable improvement in at least one of evidence support, calibration, schema validity, or resource efficiency.
+6. THE Initial production target SHALL achieve at least 60 percent Fast_Path coverage on the representative corpus while meeting accuracy gates.
+7. THE GPU-seconds per accepted document SHALL improve by at least 2x relative to the current 9B-every-document baseline before full promotion.
+8. THE v3 pipeline SHALL run in Shadow_Mode for a configurable period and minimum document count before it may influence aggregation.
+9. THE Promotion process SHALL support canary percentages, automatic rollback thresholds, and one-click reversion to the current pipeline.
+10. Backtest or paper-trading performance SHALL be reported separately from extraction correctness and SHALL not override failed correctness gates.
+
+### Requirement 17: Active Learning and Specialist Fine-Tuning
+
+**User Story:** As a model owner, I want difficult and corrected examples to improve the specialist path over time, so that fewer documents require the 9B adjudicator.
+
+#### Acceptance Criteria
+
+1. THE Active_Learning_Exporter SHALL select low-confidence, conflicting, adjudicated, and reviewer-corrected examples without exporting secrets or unauthorized content.
+2. THE Export format SHALL retain source text, spans, schema labels, relations, adjudicator decisions, reviewer corrections, and provenance.
+3. THE Training_Pipeline SHALL support fine-tuning the selected specialist extractor on the Stonks Oracle schema.
+4. EACH trained artifact SHALL be evaluated against a frozen holdout and the current production artifact.
+5. A specialist model SHALL not be promoted solely because it reduces adjudication rate; it SHALL also pass field-level correctness and calibration gates.
+6. THE Registry SHALL retain model cards containing training range, dataset version, intended use, limitations, and evaluation results.
+
+### Requirement 18: Backward-Compatible Rollout
+
+**User Story:** As a maintainer, I want to ship the new pipeline incrementally, so that existing APIs and downstream services continue operating during migration.
+
+#### Acceptance Criteria
+
+1. THE Current v2 extractor SHALL remain available behind a feature flag until v3 completes shadow and canary promotion.
+2. THE Compatibility_Adapter SHALL produce the fields required by aggregation, recommendation, validation, reporting, and API consumers.
+3. THE Database migration SHALL be additive before any destructive column or table change.
+4. THE Deployment SHALL permit per-agent, per-document-type, and percentage-based routing between v2 and v3.
+5. WHEN rollback is triggered, THE System SHALL route new work to v2 without deleting v3 audit data.
+6. THE Team SHALL remove deprecated provider branches, v2 prompt logic, and compatibility mappings only in a separately approved cleanup milestone.

.kiro/specs/intelligence-pipeline-v3/tasks.md

@@ -0,0 +1,427 @@
+# Implementation Plan: Intelligence Pipeline v3
+
+## Overview
+
+Replace the monolithic 9B model extraction pipeline with a staged, evidence-grounded multi-component architecture. CPU-first specialist services handle routine extraction, sentiment, novelty, and calibration while the existing 9B vLLM model is preserved for semantic adjudication of ambiguous cases. A capability-aware inference gateway replaces duplicated provider branches, a stock-specific impact model replaces generative self-scores, and a full shadow/canary promotion process ensures measured improvement before production influence.
+
+## Tasks
+
+- [x] 1. Rotate exposed credentials
+    - [x] 1.1 Identify every live or reusable credential in `infra/helm/stonks-oracle/values-live-math.yaml` and any other tracked files
+    - [x] 1.2 Rotate database, MinIO/object-store, Redis, broker, and market-data credentials
+    - [x] 1.3 Disable the replaced keys and review relevant access logs
+    - [x] 1.4 Remove plaintext values from the working tree without copying them into issues, PRs, logs, or spec comments
+    - [x] 1.5 Purge the values from Git history using an approved coordinated history rewrite
+    - [x] 1.6 Verify that old credentials no longer authenticate
+    - _Requirements: 1.1, 1.2, 1.6_
+
+- [x] 2. Add managed secret delivery
+    - [x] 2.1 Select External Secrets, SOPS, Sealed Secrets, or the cluster-standard mechanism
+    - [x] 2.2 Replace Helm secret values with secret references
+    - [x] 2.3 Document bootstrap and rotation procedures
+    - [x] 2.4 Add a deployment test proving pods receive required keys without values appearing in rendered manifests
+    - _Requirements: 1.3, 1.6_
+
+- [x] 3. Add repository secret scanning
+    - [x] 3.1 Add a secret scanner to pre-commit or Kiro hooks
+    - [x] 3.2 Add the scanner to pull-request and protected-branch CI
+    - [x] 3.3 Add tests/fixtures that prove real-looking secrets fail and explicit safe fixtures pass
+    - _Requirements: 1.4, 1.5_
+
+- [x] 4. Establish the current runtime source of truth
+    - [x] 4.1 Inventory active cluster deployments, agent database records, Helm releases, and environment variables
+    - [x] 4.2 Record the actual model, quantization, vLLM version, max model length, max sequences, GPU utilization limit, and current provider for every agent
+    - [x] 4.3 Resolve the conflicting Qwen/NuExtract defaults in code and infrastructure for the baseline branch
+    - [x] 4.4 Produce `docs/intelligence-pipeline-v3/current-runtime-baseline.md` without credentials
+    - _Requirements: 3.9_
+
+- [x] 5. Build a baseline replay command
+    - [x] 5.1 Add a CLI that replays a fixed document set through the current pipeline without writing trading outputs
+    - [x] 5.2 Capture structured output, schema validity, retries, duration, token usage, GPU metrics, provider/model lineage, and current downstream mappings
+    - [x] 5.3 Pin all baseline configuration and random seeds that the provider supports
+    - [x] 5.4 Store baseline reports under a versioned artifact path
+    - _Requirements: 16.2, 16.3_
+
+- [x] 6. Define the v3 annotation schema
+    - [x] 6.1 Define labels for entities, canonical companies, events, relations, numeric facts, periods, sentiment, evidence spans, direct effects, inferred exposure, and ambiguity
+    - [x] 6.2 Define evidence and adjudication guidelines with positive and negative examples
+    - [x] 6.3 Define which fields are safety-critical for promotion gates
+    - [x] 6.4 Add schema validators and sample annotations
+    - _Requirements: 16.1, 16.4_
+
+- [x] 7. Create the first Gold Corpus
+    - [x] 7.1 Sample at least 1,000 documents stratified by type, event class, length, source, company count, and difficulty
+    - [x] 7.2 Include duplicate stories, long filings, transcripts, contradictory reports, macro events, and opposing multi-company effects
+    - [x] 7.3 Double-review a hard-case subset and calculate inter-annotator agreement
+    - [x] 7.4 Freeze a holdout split that cannot be used for prompt or model tuning
+    - _Requirements: 16.1_
+
+- [x] 8. Implement evaluation metrics
+    - [x] 8.1 Implement entity/ticker precision, recall, F1, and ambiguity accuracy
+    - [x] 8.2 Implement event and relation macro/micro F1
+    - [x] 8.3 Implement numeric exact/tolerance-aware matching
+    - [x] 8.4 Implement evidence offset validity and support rate
+    - [x] 8.5 Implement sentiment macro-F1 and probability calibration metrics
+    - [x] 8.6 Implement latency, throughput, token, CPU, GPU, and memory metrics
+    - [x] 8.7 Generate per-document-type and per-difficulty reports
+    - _Requirements: 16.3, 16.4_
+
+- [x] 9. Benchmark corrected current-model extraction
+    - [x] 9.1 Run the current request unchanged
+    - [x] 9.2 Run the same 9B model with temperature zero
+    - [x] 9.3 Run the same 9B model with strict JSON Schema output and temperature zero
+    - [x] 9.4 Quantify how much of the apparent architecture gain comes from fixing the current request alone
+    - _Requirements: 16.2, 16.3, 16.5_
+
+- [x] 10. Add shared inference domain models
+    - [x] 10.1 Create `services/shared/inference/models.py` with capabilities, target, request, result, usage, and lineage types
+    - [x] 10.2 Create normalized error categories for timeout, authentication, rate limit, server, invalid response, schema, capability, and policy failures
+    - [x] 10.3 Add serialization tests proving credentials and sensitive headers are excluded
+    - _Requirements: 2.1, 2.8, 2.9_
+
+- [x] 11. Implement `OpenAICompatibleClient`
+    - [x] 11.1 Implement `/v1/chat/completions` using `httpx.AsyncClient`
+    - [x] 11.2 Implement Bearer and configurable authentication headers via runtime secret resolution
+    - [x] 11.3 Implement standard `response_format.json_schema` payloads
+    - [x] 11.4 Implement configurable vLLM `structured_outputs` extra-body payloads
+    - [x] 11.5 Implement explicit JSON-object and prompt-only fallback policies
+    - [x] 11.6 Capture request ID, usage, finish reason, structured mode, retries, and provider error category
+    - [x] 11.7 Revalidate parsed JSON locally against the supplied schema
+    - [x] 11.8 Add contract tests against a mocked compatible server and the cluster vLLM deployment
+    - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.7, 2.9_
+
+- [x] 12. Refactor Ollama support into `OllamaNativeClient`
+    - [x] 12.1 Move current Ollama request logic behind the shared request/result types
+    - [x] 12.2 Honor configured max output tokens and context settings consistently
+    - [x] 12.3 Preserve native schema formatting when supported and explicitly report prompt-only mode otherwise
+    - [x] 12.4 Retain stall/loop detection as Ollama-specific policy without leaking it into the generic protocol
+    - _Requirements: 2.1, 2.6_
+
+- [x] 13. Implement capability probing
+    - [x] 13.1 Probe health and model listing
+    - [x] 13.2 Probe strict JSON Schema with a minimal schema
+    - [x] 13.3 Probe usage metadata, seed behavior, and output-token field compatibility
+    - [x] 13.4 Store probe results and software/version metadata with TTL
+    - [x] 13.5 Refuse activation when declared required capabilities fail
+    - _Requirements: 2.10, 2.11_
+
+- [x] 14. Replace provider fallback behavior
+    - [x] 14.1 Replace `VLLMClient` with an alias/profile using `OpenAICompatibleClient`
+    - [x] 14.2 Make unknown providers a typed configuration error
+    - [x] 14.3 Add migration warnings for `vllm` provider records
+    - [x] 14.4 Add property tests proving unknown providers never invoke Ollama
+    - _Requirements: 2.2, 2.6_
+
+- [x] 15. Migrate all LLM consumers
+    - [x] 15.1 Migrate document extraction
+    - [x] 15.2 Migrate global event classification
+    - [x] 15.3 Migrate thesis rewriting and remove duplicate provider branching
+    - [x] 15.4 Replace direct/private `_config` mutation with an explicit target refresh or client-pool lifecycle
+    - [x] 15.5 Fix persistence so actual endpoint, model, and route lineage are recorded
+    - _Requirements: 2.12, 13.6_
+
+- [x] 16. Add registry migrations
+    - [x] 16.1 Create `inference_endpoints`
+    - [x] 16.2 Create `model_deployments`
+    - [x] 16.3 Create `agent_stage_bindings`
+    - [x] 16.4 Add revision, audit, uniqueness, and enabled-state constraints
+    - [x] 16.5 Add additive lineage columns/tables for existing performance logs
+    - _Requirements: 3.1, 3.2, 3.3, 3.4_
+
+- [x] 17. Implement registry resolver
+    - [x] 17.1 Resolve active stage bindings with TTL caching
+    - [x] 17.2 Invalidate cache on revisions and failed probes
+    - [x] 17.3 Resolve authentication only at invocation time
+    - [x] 17.4 Add deterministic resolution and fail-closed tests
+    - _Requirements: 3.5, 3.9_
+
+- [x] 18. Migrate existing provider records
+    - [x] 18.1 Create the current Ollama endpoint profile if in use
+    - [x] 18.2 Create the current vLLM OpenAI-compatible endpoint profile
+    - [x] 18.3 Create model deployments matching actual runtime state
+    - [x] 18.4 Convert agent and variant provider/model fields to stage bindings while retaining compatibility reads
+    - [x] 18.5 Remove conflicting runtime model defaults after migration verification
+    - _Requirements: 3.8, 3.9_
+
+- [x] 19. Add endpoint API and UI
+    - [x] 19.1 Add CRUD endpoints that never return secret values
+    - [x] 19.2 Add probe, enable, disable, and test-structured-output actions
+    - [x] 19.3 Replace free-text provider inputs with protocol, endpoint, and model-deployment selectors
+    - [x] 19.4 Display last probe, capabilities, model limits, and active stage bindings
+    - [x] 19.5 Require confirmation for external endpoint egress enablement
+    - _Requirements: 3.6, 3.7_
+
+- [x] 20. Add v3 persistence tables
+    - [x] 20.1 Add pipeline runs and stage runs
+    - [x] 20.2 Add document chunks and evidence spans
+    - [x] 20.3 Add extracted entities, facts, relations, and rejected candidates
+    - [x] 20.4 Add company signal candidates and probability distributions
+    - [x] 20.5 Add adjudication decisions, routing reasons, calibration references, and model lineage
+    - [x] 20.6 Add idempotency and immutable-revision constraints
+    - _Requirements: 13.1, 13.2, 13.3_
+
+- [x] 21. Implement sentence-aware segmenter
+    - [x] 21.1 Preserve source offsets and checksums
+    - [x] 21.2 Add document-type-specific chunk strategies
+    - [x] 21.3 Preserve filing sections and transcript speakers
+    - [x] 21.4 Mark boilerplate and duplicate chunks
+    - [x] 21.5 Remove the 8,000-character truncation from v3
+    - [x] 21.6 Add property tests proving every chunk/evidence span maps exactly to source text
+    - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
+
+- [x] 22. Implement compatibility adapter skeleton
+    - [x] 22.1 Map approved v3 records to current intelligence and impact data classes
+    - [x] 22.2 Persist `hybrid` lineage plus stage details
+    - [x] 22.3 Add golden mapping tests for every legacy enum and field range
+    - [x] 22.4 Keep adapter output disabled outside replay/shadow mode
+    - _Requirements: 13.4, 13.5, 18.2_
+
+- [x] 23. Implement deterministic financial parsing
+    - [x] 23.1 Parse tickers, currencies, money, percentages, basis points, ranges, EPS, revenue, dates, and fiscal periods
+    - [x] 23.2 Store literal and normalized representations
+    - [x] 23.3 Link each candidate to exact offsets
+    - [x] 23.4 Add broad property tests for numeric formatting and unit conversions
+    - _Requirements: 5.1, 5.7, 5.8_
+
+- [x] 24. Integrate symbol registry resolution
+    - [x] 24.1 Build canonical alias indexes from existing companies and symbol registry data
+    - [x] 24.2 Return ranked candidates and ambiguity margins
+    - [x] 24.3 Separate explicit mentions from inferred exposures
+    - [x] 24.4 Preserve unresolved literal entities without invented tickers
+    - [x] 24.5 Add tests for aliases shared by multiple companies
+    - _Requirements: 5.2, 5.3, 5.4, 5.5, 5.6_
+
+- [x] 25. Create specialist inference service
+    - [x] 25.1 Add typed batch endpoints for entities, classification, relations, and structured extraction
+    - [x] 25.2 Integrate pinned GLiNER2 Large as the initial candidate
+    - [x] 25.3 Return spans, scores, model version, and schema version
+    - [x] 25.4 Add bounded dynamic batching and warm-up
+    - [x] 25.5 Add Kubernetes CPU deployment, health probes, and metrics
+    - [x] 25.6 Add contract and load tests
+    - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.7, 6.9_
+
+- [x] 26. Integrate company-specific sentiment
+    - [x] 26.1 Build company-linked evidence groups
+    - [x] 26.2 Integrate pinned FinBERT baseline
+    - [x] 26.3 Store full probability distributions and evidence IDs
+    - [x] 26.4 Implement mixed sentiment from evidence-group disagreement
+    - [x] 26.5 Benchmark and calibrate on the Gold_Corpus
+    - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7_
+
+- [x] 27. Benchmark NuExtract 1.5 Smol
+    - [x] 27.1 Add an isolated adapter and CPU/on-demand deployment
+    - [x] 27.2 Test hierarchical extraction on long filings and transcripts
+    - [x] 27.3 Measure incremental correctness over GLiNER2 plus deterministic parsing
+    - [x] 27.4 Measure CPU latency and memory
+    - [x] 27.5 Promote it only for document classes where incremental value passes a predefined gate
+    - _Requirements: 6.6_
+
+- [x] 28. Add evidence verification
+    - [x] 28.1 Validate offsets, source text, entity association, and numeric consistency
+    - [x] 28.2 Add rejected-candidate storage and reason codes
+    - [x] 28.3 Benchmark a compact entailment verifier on claims exact matching cannot validate
+    - [x] 28.4 Add unsupported-claim and evidence-coverage metrics
+    - _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7_
+
+- [x] 29. Implement retrieval-based novelty
+    - [x] 29.1 Add exact and near-duplicate fingerprints
+    - [x] 29.2 Add replaceable compact embedding backend
+    - [x] 29.3 Index document and canonical company-event embeddings
+    - [x] 29.4 Return nearest matches and similarity scores
+    - [x] 29.5 Implement and version the novelty formula
+    - [x] 29.6 Compare novelty values against human duplicate/novelty labels
+    - _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7_
+
+- [x] 30. Build confidence feature pipeline
+    - [x] 30.1 Compute field-level features from extraction, resolution, evidence, sentiment, and agreement
+    - [x] 30.2 Train and compare calibration methods on training folds
+    - [x] 30.3 Evaluate ECE and Brier score on held-out data
+    - [x] 30.4 Version and load calibration artifacts
+    - [x] 30.5 Define conservative defaults for underrepresented classes
+    - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.6, 10.7_
+
+- [x] 31. Implement deterministic routing engine
+    - [x] 31.1 Define routing reason enums
+    - [x] 31.2 Implement hard ambiguity/conflict rules
+    - [x] 31.3 Implement calibrated fast-path thresholds by document and event type
+    - [x] 31.4 Store every route decision and feature snapshot
+    - [x] 31.5 Add property tests for determinism and threshold boundaries
+    - _Requirements: 10.5, 11.6_
+
+- [x] 32. Define adjudication schemas
+    - [x] 32.1 Define candidate, conflict, question, evidence, and decision models
+    - [x] 32.2 Exclude authoritative confidence, novelty, impact, and horizon from the model output
+    - [x] 32.3 Require evidence IDs for every material decision
+    - _Requirements: 11.2, 11.5, 11.7_
+
+- [x] 33. Build focused adjudication prompts
+    - [x] 33.1 Build packets from only relevant chunks and candidates
+    - [x] 33.2 Use strict JSON Schema and temperature zero
+    - [x] 33.3 Set a bounded output budget appropriate to decisions rather than long summaries
+    - [x] 33.4 Add prompt/version metadata and exact provider lineage
+    - _Requirements: 11.2, 11.3, 11.4_
+
+- [x] 34. Deploy and validate the 9B adjudicator
+    - [x] 34.1 Pin the approved 9B model and vLLM version
+    - [x] 34.2 Verify strict structured output with the deployment's actual vLLM version
+    - [x] 34.3 Measure peak VRAM against the current baseline and enforce the +5 percent gate
+    - [x] 34.4 Load-test concurrency and select a safe application semaphore
+    - [x] 34.5 Add availability and queue-depth alerts
+    - _Requirements: 11.1, 11.10_
+
+- [x] 35. Add post-adjudication verification
+    - [x] 35.1 Verify every referenced evidence ID was included in the packet
+    - [x] 35.2 Reject unsupported or schema-incompatible decisions
+    - [x] 35.3 Preserve both pre-adjudication candidates and final decisions
+    - [x] 35.4 Route repeated failures to review rather than accepting repaired defaults
+    - _Requirements: 11.8, 8.7_
+
+- [x] 36. Define event-time feature snapshots
+    - [x] 36.1 Define feature names, types, timing rules, and missing-value policy
+    - [x] 36.2 Include event, sentiment, magnitude, surprise, source, novelty, evidence, company, volatility, volume, and regime features
+    - [x] 36.3 Persist immutable feature snapshots at prediction time
+    - [x] 36.4 Add leakage tests preventing post-event data from entering features
+    - _Requirements: 12.2, 12.10_
+
+- [x] 37. Build outcome labels
+    - [x] 37.1 Define approved market benchmarks and abnormal-return calculations
+    - [x] 37.2 Generate signed and absolute response labels for intraday, 1d, 7d, 30d, and 90d horizons
+    - [x] 37.3 Generate abnormal-volume and time-to-peak labels where data quality permits
+    - [x] 37.4 Version label-generation code and market-data snapshots
+    - _Requirements: 12.3_
+
+- [x] 38. Implement deterministic impact baseline
+    - [x] 38.1 Map event classes, sentiment, magnitude, evidence, novelty, and source credibility to conservative outputs
+    - [x] 38.2 Unit-test every event type and boundary
+    - [x] 38.3 Use this baseline whenever no approved trained model exists
+    - _Requirements: 12.4, 12.8_
+
+- [x] 39. Train calibrated tabular impact models
+    - [x] 39.1 Train CPU-efficient gradient-boosted candidates for direction, magnitude, and horizon
+    - [x] 39.2 Use walk-forward/out-of-time splits
+    - [x] 39.3 Calibrate probabilities on a separate calibration fold
+    - [x] 39.4 Report metrics by event, sector, market cap, source, and regime
+    - [x] 39.5 Register artifacts, feature versions, training ranges, and model cards
+    - _Requirements: 12.4, 12.5, 12.6, 12.10_
+
+- [x] 40. Integrate impact outputs
+    - [x] 40.1 Store full direction, magnitude, horizon, and uncertainty outputs
+    - [x] 40.2 Map approved outputs to legacy `impact_score` and `impact_horizon` through the compatibility adapter
+    - [x] 40.3 Remove generative impact/novelty/confidence from aggregation inputs in v3 mode
+    - [x] 40.4 Add comparison dashboards against realized outcomes
+    - _Requirements: 12.1, 12.7, 12.9_
+
+- [x] 41. Implement v3 pipeline orchestrator
+    - [x] 41.1 Create explicit stage state transitions and idempotency keys
+    - [x] 41.2 Add fast-path, adjudication, persistence, and review queues
+    - [x] 41.3 Implement leases, retry policies, dead-letter handling, and resumable stages
+    - [x] 41.4 Keep v2 and v3 routing behind independent feature flags
+    - _Requirements: 14.1, 14.2, 14.5, 14.8_
+
+- [x] 42. Add bounded application parallelism
+    - [x] 42.1 Replace the single sequential extraction loop for v3 with configurable async workers
+    - [x] 42.2 Add specialist micro-batching
+    - [x] 42.3 Add adjudicator semaphore and queue backpressure
+    - [x] 42.4 Add load shedding rules that never drop safety-critical documents silently
+    - _Requirements: 14.1, 14.3, 14.4, 14.5_
+
+- [x] 43. Add traces and metrics
+    - [x] 43.1 Trace every stage under one document trace ID
+    - [x] 43.2 Add stage latency, errors, batch size, queue depth, and route metrics
+    - [x] 43.3 Add field accuracy, evidence coverage, calibration, fast-path rate, and adjudication reason dashboards
+    - [x] 43.4 Add GPU memory, utilization, and GPU-seconds per document
+    - [x] 43.5 Add alerts for schema failures, unsupported claims, calibration drift, queue saturation, and provider probe failures
+    - _Requirements: 14.7, 15.1, 15.2, 15.5_
+
+- [x] 44. Add audit/review API and UI
+    - [x] 44.1 Display source evidence and offsets for each fact
+    - [x] 44.2 Display specialist probabilities, routing reasons, adjudicator decisions, and impact-model outputs separately
+    - [x] 44.3 Allow immutable reviewer correction events
+    - [x] 44.4 Add filters for low confidence, unsupported claims, and adjudicated documents
+    - _Requirements: 15.3, 15.4, 15.6, 15.7_
+
+- [x] 45. Run offline replay
+    - [x] 45.1 Compare every required system configuration on the Gold_Corpus
+    - [x] 45.2 Publish field-level, calibration, resource, and difficulty-bucket reports
+    - [x] 45.3 Confirm corrected current-9B baseline versus full v3 incremental gain
+    - [x] 45.4 Reject or retune any stage failing safety-critical gates
+    - _Requirements: 16.2, 16.3, 16.4, 16.5_
+
+- [x] 46. Enable production shadow mode
+    - [x] 46.1 Run v3 for live documents without affecting aggregation or trading
+    - [x] 46.2 Compare v2/v3 disagreements and sample reviews by risk
+    - [x] 46.3 Measure fast-path coverage, GPU reduction, and operational stability
+    - [x] 46.4 Require the configured minimum shadow duration and document count
+    - _Requirements: 16.6, 16.7, 16.8_
+
+- [x] 47. Canary compatibility outputs
+    - [x] 47.1 Enable v3 adapter outputs for non-trading consumers first
+    - [x] 47.2 Add percentage- and document-type-based routing
+    - [x] 47.3 Configure automatic rollback on correctness, latency, queue, or availability thresholds
+    - [x] 47.4 Verify rollback leaves v3 audit records intact
+    - _Requirements: 16.9, 18.4, 18.5_
+
+- [x] 48. Canary signal influence
+    - [x] 48.1 Enable v3 signals in paper trading at a small percentage
+    - [x] 48.2 Report extraction correctness separately from trading outcomes
+    - [x] 48.3 Review material recommendation divergences
+    - [x] 48.4 Promote only after explicit owner approval and all gates pass
+    - _Requirements: 16.9, 16.10_
+
+- [x] 49. Build active-learning export
+    - [x] 49.1 Select low-confidence, conflicting, adjudicated, and corrected cases
+    - [x] 49.2 Remove or policy-filter sensitive content
+    - [x] 49.3 Export source spans, labels, relations, decisions, and provenance in a versioned format
+    - _Requirements: 17.1, 17.2_
+
+- [x] 50. Fine-tune specialist extractor
+    - [x] 50.1 Train GLiNER2 candidate artifacts on the Stonks Oracle schema
+    - [x] 50.2 Evaluate against frozen holdout and production artifact
+    - [x] 50.3 Calibrate new scores and update routing thresholds
+    - [x] 50.4 Promote only when correctness gates pass, not merely when adjudication rate falls
+    - _Requirements: 17.3, 17.4, 17.5, 17.6_
+
+- [x] 51. Deprecate legacy paths
+    - [x] 51.1 Remove duplicated `VLLMClient`/provider branching after all consumers use the gateway
+    - [x] 51.2 Remove v2 8,000-character truncation and monolithic extraction prompt after v2 retirement
+    - [x] 51.3 Remove obsolete environment/model defaults and provider free-text fields
+    - [x] 51.4 Remove compatibility adapter only after every downstream consumer reads v3 natively
+    - [x] 51.5 Archive final migration and benchmark reports
+    - _Requirements: 18.6_
+
+- [x] 52. Checkpoint — Ensure all tests pass
+    - Ensure all tests pass, ask the user if questions arise.
+
+## Task Dependency Graph
+
+```json
+{
+  "waves": [
+    { "id": 0, "tasks": ["1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "2.1", "2.2", "2.3", "2.4", "3.1", "3.2", "3.3", "4.1", "4.2", "4.3", "4.4", "5.1", "5.2", "5.3", "5.4"] },
+    { "id": 1, "tasks": ["6.1", "6.2", "6.3", "6.4", "7.1", "7.2", "7.3", "7.4", "8.1", "8.2", "8.3", "8.4", "8.5", "8.6", "8.7", "9.1", "9.2", "9.3", "9.4"] },
+    { "id": 2, "tasks": ["10.1", "10.2", "10.3", "11.1", "11.2", "11.3", "11.4", "11.5", "11.6", "11.7", "11.8", "12.1", "12.2", "12.3", "12.4"] },
+    { "id": 3, "tasks": ["13.1", "13.2", "13.3", "13.4", "13.5", "14.1", "14.2", "14.3", "14.4", "15.1", "15.2", "15.3", "15.4", "15.5", "16.1", "16.2", "16.3", "16.4", "16.5"] },
+    { "id": 4, "tasks": ["17.1", "17.2", "17.3", "17.4", "18.1", "18.2", "18.3", "18.4", "18.5", "19.1", "19.2", "19.3", "19.4", "19.5", "20.1", "20.2", "20.3", "20.4", "20.5", "20.6", "21.1", "21.2", "21.3", "21.4", "21.5", "21.6", "22.1", "22.2", "22.3", "22.4"] },
+    { "id": 5, "tasks": ["23.1", "23.2", "23.3", "23.4", "24.1", "24.2", "24.3", "24.4", "24.5", "25.1", "25.2", "25.3", "25.4", "25.5", "25.6", "26.1", "26.2", "26.3", "26.4", "26.5", "27.1", "27.2", "27.3", "27.4", "27.5", "28.1", "28.2", "28.3", "28.4"] },
+    { "id": 6, "tasks": ["29.1", "29.2", "29.3", "29.4", "29.5", "29.6", "30.1", "30.2", "30.3", "30.4", "30.5", "31.1", "31.2", "31.3", "31.4", "31.5"] },
+    { "id": 7, "tasks": ["32.1", "32.2", "32.3", "33.1", "33.2", "33.3", "33.4", "34.1", "34.2", "34.3", "34.4", "34.5", "35.1", "35.2", "35.3", "35.4"] },
+    { "id": 8, "tasks": ["36.1", "36.2", "36.3", "36.4", "37.1", "37.2", "37.3", "37.4", "38.1", "38.2", "38.3", "39.1", "39.2", "39.3", "39.4", "39.5", "40.1", "40.2", "40.3", "40.4"] },
+    { "id": 9, "tasks": ["41.1", "41.2", "41.3", "41.4", "42.1", "42.2", "42.3", "42.4", "43.1", "43.2", "43.3", "43.4", "43.5", "44.1", "44.2", "44.3", "44.4"] },
+    { "id": 10, "tasks": ["45.1", "45.2", "45.3", "45.4", "46.1", "46.2", "46.3", "46.4", "47.1", "47.2", "47.3", "47.4", "48.1", "48.2", "48.3", "48.4"] },
+    { "id": 11, "tasks": ["49.1", "49.2", "49.3", "50.1", "50.2", "50.3", "50.4", "51.1", "51.2", "51.3", "51.4", "51.5"] }
+  ]
+}
+```
+
+## Notes
+
+- This plan is intentionally staged so the current system remains available until the replacement is measured and promoted
+- Task 1 (credential rotation) is a **BLOCKER** — must complete before any feature deployment
+- Tasks within the same wave may run in parallel; tasks in later waves depend on earlier waves completing
+- Each phase ends with an explicit evidence artifact: test output, benchmark report, migration result, or deployment probe
+- The current v2 extractor remains available behind a feature flag until v3 completes shadow and canary promotion
+- Peak GPU memory must not exceed the measured current 9B deployment baseline by more than 5 percent
+- The definition of done requires: credentials rotated, all consumers on shared gateway, evidence-linked outputs, calibrated scores replacing generative self-scores, 9B preserved for adjudication, and shadow/canary gates passed
+- Rollback to the current production path must be exercised successfully before full promotion
+- Property tests validate deterministic behaviors (unknown providers fail closed, chunk offsets map to source, routing thresholds are deterministic)
+- Deprecated legacy paths (task 51) require separate approval and must not proceed until all downstream consumers read v3 natively

.kiro/specs/math-core-v3-engine/.config.kiro

@@ -0,0 +1 @@
+{"specId": "ff0d03d7-3469-4551-bf05-15295b735c83", "workflowType": "requirements-first", "specType": "feature"}

.kiro/specs/math-core-v3-engine/design.md

@@ -0,0 +1,851 @@
+# Design Document: Math Core v3 Engine
+
+## Overview
+
+The v3 Calibrated Evidence Engine replaces the current dual-mode pipeline (heuristic + probabilistic) with a principled Bayesian evidence accumulation system. The upgrade transforms the signal processing core from weighted-sentiment averaging to:
+
+```
+EvidenceUnit → calibrated reliability (q_i) → log-likelihood ratio (LLR_i)
+→ correlation-adjusted cluster LLR → posterior P_up → return distribution → EV/risk decision
+```
+
+**Key design goals:**
+- Replace arbitrary weight products with calibrated probabilistic evidence
+- Prevent correlated articles from inflating evidence counts
+- Make confidence multiplicative so one bad dimension suppresses the trade
+- Size positions using fractional Kelly criterion under hard risk caps
+- Preserve the existing three-layer architecture and service boundaries
+- Gate behind `v3_engine_enabled` feature flag with heuristic fallback
+
+**What changes vs. what stays:**
+- The `WeightedSignal` abstraction remains as an intermediate before LLR conversion
+- All output goes into existing JSONB metadata columns (no new migrations)
+- Service file boundaries are preserved; internals are upgraded
+- The heuristic pipeline stays as fallback, controlled by a single flag read per aggregation cycle
+
+## Architecture
+
+```mermaid
+flowchart TD
+    subgraph Input Layer
+        CS[Company Signals]
+        MS[Macro Signals]
+        XS[Competitive Signals]
+    end
+
+    subgraph Normalization
+        EU[EvidenceUnit Normalization]
+    end
+
+    subgraph Scoring Pipeline
+        QI[Calibrated Reliability q_i]
+        LLR[LLR Conversion]
+    end
+
+    subgraph Clustering
+        CL[Correlation-Aware Clustering]
+        NEFF[n_eff Computation]
+        CLLR[Cluster LLR]
+    end
+
+    subgraph Posterior Assembly
+        REG[Regime Detection v3]
+        POST[Log-Odds Posterior P_up]
+        CON[LLR Entropy Contradiction]
+        CONF[Multiplicative Confidence]
+    end
+
+    subgraph Decision Layer
+        PROJ[Posterior State Projection]
+        RET[Return Distribution / EV Gate]
+        ELIG[Regime-Aware Eligibility]
+        KELLY[Fractional Kelly Sizing]
+        STOP[Regime-Aware Stops]
+        HEAT[Stop-Defined Portfolio Heat]
+    end
+
+    subgraph Quality & Control
+        DQ[Data Quality v3]
+        FF[Feature Flag Router]
+        TIER[Risk Tier Auto-Adjustment]
+    end
+
+    CS --> EU
+    MS --> EU
+    XS --> EU
+    EU --> QI
+    QI --> LLR
+    LLR --> CL
+    CL --> NEFF
+    NEFF --> CLLR
+    REG --> POST
+    CLLR --> POST
+    POST --> CON
+    POST --> CONF
+    CONF --> PROJ
+    PROJ --> RET
+    RET --> ELIG
+    ELIG --> KELLY
+    KELLY --> STOP
+    KELLY --> HEAT
+    DQ --> CONF
+    FF --> EU
+    TIER --> KELLY
+```
+
+### Service File Mapping
+
+| Service File | v3 Responsibility |
+|---|---|
+| `services/aggregation/scoring.py` | EvidenceUnit normalization, q_i pipeline, LLR conversion |
+| `services/aggregation/bayesian.py` | Posterior assembly via log-odds, P_up, strength |
+| `services/aggregation/contradiction.py` | LLR entropy contradiction score |
+| `services/aggregation/regime.py` | Regime detection v3 (ATR-normalized trend_z) |
+| `services/aggregation/interpolation.py` | Noisy-OR macro exposure, LLR emission |
+| `services/aggregation/signal_propagation.py` | Correlation-shrunk competitive propagation |
+| `services/aggregation/projection.py` | Posterior state A_t, regime-aware decay |
+| `services/aggregation/worker.py` | Orchestration, clustering, n_eff, confidence assembly |
+| `services/recommendation/eligibility.py` | EV gate, regime-aware eligibility, mode escalation |
+| `services/trading/position_sizer.py` | Fractional Kelly sizing under caps |
+| `services/trading/stop_loss_manager.py` | Regime-aware stop/TP, trailing activation |
+| `services/risk/engine.py` | Stop-defined heat, tier auto-adjustment |
+
+### Feature Flag Flow
+
+```mermaid
+sequenceDiagram
+    participant Worker as aggregation/worker.py
+    participant DB as risk_configs table
+    participant V3 as v3 Pipeline
+    participant Heuristic as Heuristic Pipeline
+
+    Worker->>DB: SELECT v3_engine_enabled
+    alt v3_engine_enabled = True
+        Worker->>V3: Run v3 pipeline
+        V3-->>Worker: Posterior + confidence + EV
+        alt Unhandled error
+            V3-->>Worker: Exception
+            Worker->>Heuristic: Fallback to heuristic
+            Worker->>Worker: Log error, record fallback in metadata
+        end
+    else v3_engine_enabled = False or DB error
+        Worker->>Heuristic: Run heuristic pipeline
+    end
+```
+
+## Components and Interfaces
+
+### 1. EvidenceUnit Dataclass (`scoring.py`)
+
+```python
+@dataclass(frozen=True)
+class EvidenceUnit:
+    """Canonical normalized signal representation for v3 pipeline."""
+    symbol: str
+    layer: str  # "company" | "macro" | "competitive"
+    event_type: str
+    source_id: str
+    source_group: str
+    timestamp: datetime
+    horizon: str  # "intraday" | "1d" | "7d" | "30d" | "90d"
+    direction: int  # -1, 0, +1
+    sentiment_strength: float  # [0, 1]
+    impact: float  # [0, 1]
+    extraction_conf: float  # [0, 1]
+    source_cred: float  # [0, 1]
+    novelty: float  # [0, 1]
+    event_base_rate: float  # (0, 1]
+    cluster_id: str
+```
+
+**Normalization functions** — one per layer:
+- `normalize_company_signal(impact_row, ...) -> EvidenceUnit`
+- `normalize_macro_signal(macro_impact_record, global_event, ...) -> EvidenceUnit`
+- `normalize_competitive_signal(competitive_signal_record, ...) -> EvidenceUnit`
+
+Each validates required fields (symbol, timestamp, source_id), substitutes 0.5 for missing optional numeric fields, and assigns direction from sentiment/impact_direction strings.
+
+### 2. Calibrated Reliability Pipeline (`scoring.py`)
+
+```python
+@dataclass(frozen=True)
+class ReliabilityComponents:
+    q_ext: float
+    q_source: float
+    q_recency: float
+    q_uniqueness: float
+    q_i: float  # final combined reliability
+
+def compute_v3_reliability(
+    unit: EvidenceUnit,
+    source_stats: SourceStats,
+    cluster_position: int,  # duplicate_count_before
+    reference_time: datetime,
+) -> ReliabilityComponents: ...
+```
+
+Sub-computations:
+- `q_ext = sigmoid(8.0 * (extraction_conf - 0.55))`
+- `q_source = clamp((E[theta_s] - 0.50) / 0.35, 0, 1)` with Beta(alpha_0+hits, beta_0+misses)
+- `q_recency = 2^(-age_hours / tau_adaptive)` with adaptive half-life
+- `q_uniqueness = clamp(0.5 + 0.5 * novelty, 0.5, 1.0) * (1 / sqrt(1 + dup_count))`
+- `q_i = clamp(q_ext * q_source * source_cred * q_recency * q_uniqueness, 0, 1)`
+
+### 3. LLR Conversion (`scoring.py`)
+
+```python
+def compute_llr(unit: EvidenceUnit, q_i: float) -> float:
+    """Convert calibrated reliability to log-likelihood ratio."""
+    p_correct = clamp(0.50 + 0.35 * q_i * unit.impact * unit.sentiment_strength, 0.501, 0.85)
+    if unit.direction == 0:
+        return 0.0
+    return unit.direction * math.log(p_correct / (1 - p_correct))
+```
+
+### 4. Correlation-Aware Clustering (`worker.py`)
+
+```python
+@dataclass
+class EvidenceCluster:
+    cluster_id: str
+    units: list[EvidenceUnit]
+    llrs: list[float]
+    n_eff: float
+    cluster_llr: float
+
+def cluster_evidence(units: list[EvidenceUnit], llrs: list[float]) -> list[EvidenceCluster]:
+    """Group by (symbol, horizon, event_type, source_group, time_bucket)."""
+    ...
+
+def compute_n_eff(llrs: list[float], correlations: list[list[float]]) -> float:
+    """n_eff = (sum w_i)^2 / (sum w_i^2 + 2*sum_{i<j} rho_ij*w_i*w_j)"""
+    ...
+
+def compute_cluster_llr(llrs: list[float], n_eff: float) -> float:
+    """LLR_c = clamp(weighted_mean(LLR_i, |LLR_i|) * sqrt(n_eff), -2.5, 2.5)"""
+    ...
+```
+
+Default pairwise correlations:
+| Relationship | rho |
+|---|---:|
+| Same wire/story/source group | 0.80 |
+| Same event, different publisher | 0.50 |
+| Same theme, different event | 0.25 |
+| Independent events | 0.00 |
+
+### 5. Posterior Assembly (`bayesian.py`)
+
+```python
+@dataclass(frozen=True)
+class V3Posterior:
+    p_up: float          # sigmoid(log_odds)
+    p_down: float        # 1 - p_up
+    log_odds: float      # logit(P_prior) + sum(gamma * LLR_c)
+    strength: float      # abs(2 * p_up - 1)
+    direction: str       # bullish | bearish | neutral
+    n_eff_total: float
+    regime: str
+
+def compute_v3_posterior(
+    clusters: list[EvidenceCluster],
+    regime: RegimeClassification,
+    p_prior: float = 0.50,
+) -> V3Posterior: ...
+```
+
+Direction thresholds by regime:
+| Regime | Bullish if P_up >= | Bearish if P_up <= |
+|---|---:|---:|
+| panic | 0.68 | 0.32 |
+| trend_following | 0.60 | 0.40 |
+| mean_reversion | 0.63 | 0.37 |
+| uncertainty | 0.65 | 0.35 |
+
+### 6. Regime Detection v3 (`regime.py`)
+
+```python
+@dataclass(frozen=True)
+class V3RegimeClassification:
+    regime: MarketRegime
+    trend_z: float           # (EMA_20 - EMA_100) / ATR_20
+    vol_ratio: float         # sigma_20 / sigma_100
+    evidence_multiplier: float  # gamma_regime
+    confidence_multiplier: float
+    phi_decay: float         # for projection
+    atr_multiplier: float    # for stops
+
+def classify_regime_v3(
+    closing_prices: list[float],
+    daily_returns: list[float],
+    atr_20: float,
+) -> V3RegimeClassification: ...
+```
+
+Classification rules (in priority order):
+1. **Panic**: vol_ratio > 1.5 OR abs(trend_z) > 2.5
+2. **Trend following**: abs(trend_z) >= 0.75 AND vol_ratio < 1.3
+3. **Mean reversion**: abs(trend_z) < 0.50 AND vol_ratio < 1.0
+4. **Uncertainty**: all other cases
+
+### 7. LLR Entropy Contradiction (`contradiction.py`)
+
+```python
+def compute_v3_contradiction(clusters: list[EvidenceCluster]) -> float:
+    """
+    E_pos = sum(max(LLR_c, 0))
+    E_neg = sum(max(-LLR_c, 0))
+    H = -f_pos*log2(f_pos) - f_neg*log2(f_neg)
+    volume_factor = 1 - exp(-E_total / 3.0)
+    return H * volume_factor
+    """
+    ...
+```
+
+### 8. Multiplicative Confidence (`worker.py`)
+
+```python
+def compute_v3_confidence(
+    n_eff_total: float,
+    q_values: list[float],
+    llrs: list[float],
+    strength: float,
+    regime_confidence_mult: float,
+    contradiction: float,
+    data_quality: float,
+) -> float:
+    """
+    C_evidence = 1 - exp(-n_eff_total / 5.0)
+    C_quality = weighted_mean(q_i, |LLR_i|)
+    confidence = clamp(
+        C_evidence * sqrt(C_quality) * sqrt(max(strength, 0.05))
+        * regime_confidence_mult * (1 - contradiction) * data_quality,
+        0, 1
+    )
+    """
+    ...
+```
+
+### 9. Noisy-OR Macro Exposure (`interpolation.py`)
+
+```python
+def compute_normalized_macro_exposure(overlaps: dict[str, float]) -> float:
+    """
+    E_raw = 1 - prod(1 - w_k * O_k)
+    E_max = 1 - prod(1 - w_k)
+    return E_raw / E_max
+    """
+    ...
+
+def compute_macro_llr(
+    macro_impact: float,
+    event_confidence: float,
+    q_recency: float,
+    macro_direction: int,
+) -> float:
+    """
+    p_macro = clamp(0.50 + 0.30 * macro_impact * event_confidence * q_recency, 0.501, 0.80)
+    return macro_direction * log(p_macro / (1 - p_macro))
+    """
+    ...
+```
+
+### 10. Correlation-Shrunk Competitive Propagation (`signal_propagation.py`)
+
+```python
+def compute_shrunk_correlation(
+    rho_rolling: float,
+    n_observations: int,
+    same_sector: bool,
+) -> float:
+    """
+    rho_prior = 0.30 if same_sector else 0.10
+    rho_shrunk = (n/(n+30)) * rho_rolling + (30/(n+30)) * rho_prior
+    return max(rho_shrunk, 0)
+    """
+    ...
+
+def compute_competitive_llr(
+    llr_source: float,
+    rho_effective: float,
+    d_network: int,
+    pattern_confidence: float,
+) -> float:
+    """
+    attenuation = rho_effective * exp(-0.85 * d_network)
+    return clamp(llr_source * attenuation * pattern_confidence, -1.25, 1.25)
+    """
+    ...
+```
+
+### 11. Posterior State Projection (`projection.py`)
+
+```python
+@dataclass
+class V3ProjectionState:
+    a_t: float              # accumulated evidence state
+    p_up_projected: float   # sigmoid(logit(P_prior) + phi^h * A_t)
+    projected_strength: float
+    diverges: bool
+    phi_regime: float
+
+def compute_v3_projection(
+    a_prev: float,
+    cluster_llrs: list[float],
+    regime: V3RegimeClassification,
+    p_prior: float,
+    projection_horizon: int,
+    known_catalyst_llr: float = 0.0,
+) -> V3ProjectionState: ...
+```
+
+Regime decay factors (phi):
+| Regime | phi |
+|---|---:|
+| panic | 0.35 |
+| trend_following | 0.80 |
+| mean_reversion | 0.55 |
+| uncertainty | 0.50 |
+
+### 12. Return Distribution and EV Gate (`eligibility.py`)
+
+```python
+@dataclass(frozen=True)
+class ReturnDistribution:
+    sigma_h: float     # realized_vol_20d * sqrt(horizon_days / 252)
+    mu_h: float        # tanh(A_projected / 3.0) * confidence * sigma_h
+    ev_long: float     # mu_h - costs - 0.10 * CVaR_5
+    min_edge: float    # regime-specific minimum edge
+    eligible: bool
+
+def compute_return_distribution(
+    a_projected: float,
+    confidence: float,
+    realized_vol_20d: float,
+    horizon_days: int,
+    costs: float,
+    regime: str,
+) -> ReturnDistribution: ...
+```
+
+### 13. Fractional Kelly Position Sizing (`position_sizer.py`)
+
+```python
+def compute_kelly_sizing(
+    p_win: float,       # P_up from posterior
+    b: float,           # reward ratio: clamp(1.2 + 2*conf + str - contra, 1.2, 3.0)
+    confidence: float,
+    data_quality: float,
+    contradiction: float,
+    max_position_pct: float,
+    available_caps: dict[str, float],  # sector, correlation, heat capacities
+) -> float:
+    """
+    f_kelly = (p_win * b - (1 - p_win)) / b
+    portfolio_pct = clamp(max(0, f_kelly) * 0.25 * confidence * data_quality * (1 - contradiction), 0, max_position_pct)
+    Apply min of all capacity constraints.
+    """
+    ...
+```
+
+### 14. Stop-Defined Portfolio Heat (`risk/engine.py`)
+
+```python
+def compute_portfolio_heat(
+    positions: list[OpenPosition],
+    stop_distances: dict[str, float],
+) -> float:
+    """risk_dollars = position_value * stop_distance_pct; heat = sum(risk_dollars)"""
+    ...
+
+def check_heat_capacity(
+    current_heat: float,
+    new_risk_dollars: float,
+    max_heat_pct: float,
+    portfolio_value: float,
+) -> bool: ...
+```
+
+### 15. Data Quality v3 (`worker.py`)
+
+```python
+def compute_v3_data_quality(
+    units: list[EvidenceUnit],
+    extraction_failure_rate: float,
+    age_newest_hours: float,
+    n_source_types: int,
+) -> float:
+    """
+    Q_parse = 1 - extraction_failure_rate
+    Q_conf = weighted_mean(extraction_conf, impact)
+    Q_fresh = exp(-age_newest_hours / 168)
+    Q_coverage = 1 - exp(-N_valid / 5)
+    Q_diversity = min(1, log2(1 + N_source_types) / log2(4))
+    return clamp(Q_parse * sqrt(Q_conf) * Q_fresh * Q_coverage * Q_diversity, 0, 1)
+    """
+    ...
+```
+
+### 16. Risk Tier Auto-Adjustment (`risk/engine.py`)
+
+```python
+@dataclass
+class TierMetrics:
+    profit_factor_30d: float
+    max_drawdown_30d: float
+    calibration_error: float
+    realized_sharpe_30d: float
+    n_trades_30d: int
+    reserve_pool_pct: float
+
+def evaluate_tier_adjustment(metrics: TierMetrics) -> str:
+    """Returns 'upgrade' | 'downgrade' | 'hold'"""
+    ...
+```
+
+## Data Models
+
+### EvidenceUnit (frozen dataclass)
+
+| Field | Type | Range | Source |
+|---|---|---|---|
+| symbol | str | — | Required from signal |
+| layer | str | company/macro/competitive | Set during normalization |
+| event_type | str | — | catalyst_type or impact_type |
+| source_id | str | — | document_id or event_id |
+| source_group | str | — | publisher / "macro" / "competitive" |
+| timestamp | datetime | — | published_at |
+| horizon | str | intraday/1d/7d/30d/90d | window or estimated_duration mapping |
+| direction | int | -1, 0, +1 | sentiment/direction mapping |
+| sentiment_strength | float | [0, 1] | impact_score or sentiment confidence |
+| impact | float | [0, 1] | impact_score |
+| extraction_conf | float | [0, 1] | confidence field |
+| source_cred | float | [0, 1] | source_credibility |
+| novelty | float | [0, 1] | novelty_score |
+| event_base_rate | float | (0, 1] | lookup by event_type |
+| cluster_id | str | — | computed hash of clustering key |
+
+### V3 Posterior Output (stored in JSONB metadata)
+
+```json
+{
+  "v3_posterior": {
+    "p_up": 0.64,
+    "p_down": 0.36,
+    "log_odds": 0.58,
+    "strength": 0.28,
+    "confidence": 0.61,
+    "contradiction": 0.18,
+    "n_eff": 5.7,
+    "data_quality": 0.82,
+    "regime": "trend_following"
+  },
+  "v3_return_model": {
+    "mu_h": 0.012,
+    "sigma_h": 0.041,
+    "ev_long": 0.007,
+    "min_edge": 0.0035
+  },
+  "v3_explainability": {
+    "top_positive_clusters": [...],
+    "top_negative_clusters": [...],
+    "suppression_reasons": [],
+    "risk_adjustments": []
+  },
+  "pipeline_mode": "v3"
+}
+```
+
+### Source Statistics (for q_source computation)
+
+```python
+@dataclass
+class SourceStats:
+    source_id: str
+    hits: int = 0      # correct directional predictions
+    misses: int = 0    # incorrect directional predictions
+    alpha_0: int = 3   # prior
+    beta_0: int = 3    # prior
+```
+
+### Regime Parameters Table
+
+| Regime | gamma (evidence) | confidence_mult | phi (decay) | ATR_mult (stops) | min_edge |
+|---|---:|---:|---:|---:|---:|
+| panic | 0.70 | 0.70 | 0.35 | 2.5 | 0.0100 |
+| trend_following | 1.10 | 1.00 | 0.80 | 1.8 | 0.0035 |
+| mean_reversion | 0.90 | 0.95 | 0.55 | 1.4 | 0.0050 |
+| uncertainty | 0.80 | 0.85 | 0.50 | 2.0 | 0.0075 |
+
+### Eligibility Thresholds (Regime-Specific)
+
+| Regime | confidence_min | contradiction_max | strength_min |
+|---|---:|---:|---:|
+| panic | 0.70 | 0.25 | 0.36 |
+| trend_following | 0.55 | 0.40 | 0.20 |
+| mean_reversion | 0.60 | 0.35 | 0.26 |
+| uncertainty | 0.65 | 0.30 | 0.30 |
+
+
+## Correctness Properties
+
+*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
+
+### Property 1: Reliability q_i is bounded in [0, 1]
+
+*For any* valid EvidenceUnit with extraction_conf in [0,1], source_cred in [0,1], novelty in [0,1], any non-negative age_hours, and any non-negative duplicate_count_before, the computed q_i SHALL be in the range [0.0, 1.0].
+
+**Validates: Requirements 2.8, 21.1**
+
+### Property 2: p_correct is bounded in [0.501, 0.85]
+
+*For any* valid q_i in [0, 1], impact in [0, 1], and sentiment_strength in [0, 1], the computed p_correct SHALL be in the range [0.501, 0.85].
+
+**Validates: Requirements 3.1, 21.2**
+
+### Property 3: LLR sign matches direction and magnitude is bounded
+
+*For any* valid signal with direction in {-1, +1}, the computed LLR SHALL have the same sign as direction, with absolute magnitude in [ln(0.501/0.499), ln(0.85/0.15)] ≈ [0.004, 1.735].
+
+**Validates: Requirements 3.2, 3.4, 3.5, 3.6, 21.3**
+
+### Property 4: Neutral signals produce zero LLR
+
+*For any* valid EvidenceUnit with direction = 0, regardless of all other field values, the computed LLR SHALL be exactly 0.0.
+
+**Validates: Requirements 1.5, 3.3**
+
+### Property 5: Effective evidence count n_eff is bounded by cluster size
+
+*For any* cluster of N signals with non-negative pairwise correlations rho_ij in [0, 1], the computed n_eff SHALL satisfy 0 < n_eff <= N.
+
+**Validates: Requirements 4.2, 21.4**
+
+### Property 6: Cluster LLR is clamped to [-2.5, 2.5]
+
+*For any* cluster configuration with any number of signals and any LLR values, the computed cluster LLR_c SHALL be in the range [-2.5, 2.5].
+
+**Validates: Requirements 4.4, 4.5**
+
+### Property 7: Posterior P_up is in open interval (0, 1)
+
+*For any* set of cluster LLRs (each in [-2.5, 2.5]), any regime evidence multiplier gamma in {0.70, 0.80, 0.90, 1.10}, and any prior P_prior in [0.40, 0.60], the computed P_up SHALL be in (1e-10, 1 - 1e-10).
+
+**Validates: Requirements 5.3, 21.5**
+
+### Property 8: Contradiction is zero when evidence is unidirectional
+
+*For any* set of cluster LLRs where all clusters have the same sign (all positive or all negative), the computed contradiction score SHALL be 0.0.
+
+**Validates: Requirements 7.7, 21.7**
+
+### Property 9: Contradiction score is bounded in [0, 1]
+
+*For any* set of cluster LLRs (including mixed positive and negative), the computed contradiction score SHALL be in the range [0.0, 1.0].
+
+**Validates: Requirements 7.6**
+
+### Property 10: Multiplicative confidence is bounded in [0, 1] and suppressed by weak dimensions
+
+*For any* valid inputs (n_eff_total >= 0, q_values in [0,1], strength in [0,1], regime_confidence_mult in (0,1], contradiction in [0,1], data_quality in [0,1]), the computed confidence SHALL be in [0, 1]. Furthermore, if any single dimension (data_quality, 1-contradiction, or C_quality) is below 0.01, the resulting confidence SHALL be below 0.10.
+
+**Validates: Requirements 8.3, 8.5, 21.6**
+
+### Property 11: Fractional Kelly sizing is bounded and respects negative edge
+
+*For any* valid inputs (P_up in (0,1), b in [1.2, 3.0], confidence in [0,1], data_quality in [0,1], contradiction in [0,1], max_position_pct > 0), the computed portfolio_pct SHALL be in [0, max_position_pct]. When f_kelly = (P_up * b - (1 - P_up)) / b <= 0, portfolio_pct SHALL be exactly 0.
+
+**Validates: Requirements 14.4, 14.7, 21.8, 21.9**
+
+### Property 12: Posterior state JSON round-trip
+
+*For any* valid V3Posterior state (p_up, log_odds, strength, confidence, contradiction, n_eff, data_quality, regime), serializing to JSON and deserializing SHALL produce an equivalent state within floating-point tolerance (1e-10).
+
+**Validates: Requirements 20.1, 21.10**
+
+### Property 13: Noisy-OR normalized exposure is bounded in [0, 1]
+
+*For any* overlap values O_k in [0, 1] for each dimension (geo, supply, commodity, sector) with fixed positive weights, the normalized macro exposure E_macro SHALL be in [0.0, 1.0], reaching exactly 1.0 when all O_k = 1.0.
+
+**Validates: Requirements 9.1, 9.2**
+
+### Property 14: Competitive LLR is clamped to [-1.25, 1.25]
+
+*For any* source LLR, shrunk correlation (non-negative), graph distance (1-3), and pattern confidence in [0,1], the computed competitive LLR SHALL be in [-1.25, 1.25].
+
+**Validates: Requirements 10.4, 10.5**
+
+### Property 15: Graph attenuation is zero beyond max distance
+
+*For any* inputs where graph distance > 3, the computed attenuation SHALL be 0.0, producing zero competitive LLR regardless of other parameters.
+
+**Validates: Requirements 10.3**
+
+### Property 16: Projection evidence state decays toward zero
+
+*For any* initial evidence state A_t and regime decay phi in (0, 1), the projected state A_projected_h = phi^h * A_t SHALL have |A_projected_h| < |A_t| for all h >= 1, converging toward 0 as h increases.
+
+**Validates: Requirements 11.1, 11.3**
+
+### Property 17: Data quality score is bounded in [0, 1]
+
+*For any* valid inputs (extraction_failure_rate in [0,1], extraction_conf_i in [0,1], impact_i in [0,1], age_newest_hours >= 0, N_valid >= 0, N_source_types >= 0), the computed data_quality_score SHALL be in [0, 1].
+
+**Validates: Requirements 17.6**
+
+### Property 18: Stop loss is below entry price and take profit is above
+
+*For any* entry_price > 0, stop_distance_pct in [0.005, 1.0), and reward ratio b >= 1.2, the computed stop_loss SHALL be less than entry_price and take_profit SHALL be greater than entry_price.
+
+**Validates: Requirements 16.2, 16.3**
+
+### Property 19: Trailing stop never decreases
+
+*For any* sequence of current prices and trailing stop computations, each new trailing_stop value SHALL be >= the previous trailing_stop value (monotonically non-decreasing).
+
+**Validates: Requirements 16.5**
+
+### Property 20: Regime classification is exhaustive and deterministic
+
+*For any* valid market data inputs (closing_prices of sufficient length, daily_returns, ATR_20 > 0), the regime classification SHALL produce exactly one of {panic, trend_following, mean_reversion, uncertainty} and the same inputs SHALL always produce the same classification.
+
+**Validates: Requirements 6.2, 6.3, 6.4, 6.5**
+
+### Property 21: EvidenceUnit normalization preserves field ranges
+
+*For any* valid company, macro, or competitive signal input, the normalized EvidenceUnit SHALL have: direction in {-1, 0, +1}, sentiment_strength in [0, 1], impact in [0, 1], extraction_conf in [0, 1], source_cred in [0, 1], novelty in [0, 1], and event_base_rate in (0, 1].
+
+**Validates: Requirements 1.1, 1.2, 1.3, 1.4, 1.7**
+
+### Property 22: Portfolio heat rejection is correct
+
+*For any* set of open positions with stop distances, if the sum of (position_value × stop_distance_pct) exceeds max_portfolio_heat × portfolio_value, then new position entry SHALL be rejected.
+
+**Validates: Requirements 15.3, 15.5**
+
+### Property 23: Tier auto-adjustment obeys downgrade-any, upgrade-all logic
+
+*For any* TierMetrics, if ANY single downgrade condition is met (profit_factor < 1.0 OR drawdown > 0.12 OR calibration_error > 0.20 OR sharpe < 0), the result SHALL be "downgrade". An "upgrade" SHALL only occur when ALL upgrade conditions are simultaneously met.
+
+**Validates: Requirements 18.3, 18.4**
+
+## Error Handling
+
+### Fail-Closed Philosophy
+
+The v3 engine follows a fail-closed design: when in doubt, suppress the trade rather than emit a false signal.
+
+| Error Scenario | Response | Fallback |
+|---|---|---|
+| `v3_engine_enabled` flag unreadable | Default to heuristic mode | Log warning |
+| Unhandled exception in v3 pipeline | Fall back to heuristic for that cycle | Log ERROR with traceback, record in metadata |
+| Missing market data for regime | Default to "uncertainty" regime | Most conservative multipliers |
+| Missing source statistics | q_source = 0.0 (neutral prior) | Source treated as untrusted |
+| Missing realized_vol_20d | Use default 0.25 annualized | Conservative volatility estimate |
+| Division by zero in n_eff | Return n_eff = 1.0 (single signal) | Denominator guard |
+| NaN/Inf in any computation | Clamp to boundary, log warning | Never propagate NaN to output |
+| data_quality < 0.50 | Force informational mode | Suppress trade recommendation |
+| No company evidence (only macro/competitive) | Force informational | Unless macro_only_enabled |
+
+### Numerical Guards
+
+All mathematical functions include:
+- **Sigmoid overflow**: Guard `exp(-x)` for x > 500 or x < -500
+- **Log domain**: Guard `log(x)` with x > 0 check; `log2(0)` treated as 0 in entropy
+- **Division by zero**: All denominators checked > 0 before division
+- **NaN propagation**: All outputs validated with `math.isnan()` check before storage
+- **Clamp boundaries**: Final values clamped to documented ranges
+
+### Graceful Degradation Chain
+
+```
+v3 pipeline error → heuristic fallback → informational mode → no recommendation
+```
+
+Each level preserves audit trail via output metadata.
+
+## Testing Strategy
+
+### Dual Testing Approach
+
+**Property-Based Tests (Hypothesis):**
+- Library: `hypothesis` (already in use in this project)
+- Configuration: `@settings(max_examples=100)` minimum per property
+- File naming: `tests/test_pbt_v3_*.py`
+- Each property test tagged with: `# Feature: math-core-v3-engine, Property N: <title>`
+- One property-based test per correctness property (23 properties → 23 PBT tests)
+
+**Unit Tests (pytest):**
+- Specific examples with known inputs/outputs for each formula
+- Edge cases: zero inputs, boundary values, NaN handling
+- Integration between components (e.g., full pipeline from EvidenceUnit to recommendation)
+- Error handling paths (DB errors, missing data, feature flag states)
+
+### Property Test Organization
+
+| Test File | Properties Covered | Module Under Test |
+|---|---|---|
+| `tests/test_pbt_v3_reliability.py` | 1, 2, 3, 4, 21 | scoring.py (q_i, p_correct, LLR) |
+| `tests/test_pbt_v3_clustering.py` | 5, 6 | worker.py (n_eff, cluster LLR) |
+| `tests/test_pbt_v3_posterior.py` | 7, 8, 9, 10, 12, 20 | bayesian.py, contradiction.py, worker.py |
+| `tests/test_pbt_v3_layers.py` | 13, 14, 15 | interpolation.py, signal_propagation.py |
+| `tests/test_pbt_v3_projection.py` | 16 | projection.py |
+| `tests/test_pbt_v3_decision.py` | 11, 17, 18, 19, 22 | position_sizer.py, stop_loss_manager.py, eligibility.py |
+| `tests/test_pbt_v3_tier.py` | 23 | risk/engine.py |
+
+### Hypothesis Strategy Design
+
+Key custom strategies for generating valid inputs:
+
+```python
+from hypothesis import strategies as st
+
+# EvidenceUnit generator
+evidence_units = st.builds(
+    EvidenceUnit,
+    symbol=st.text(min_size=1, max_size=5),
+    layer=st.sampled_from(["company", "macro", "competitive"]),
+    direction=st.sampled_from([-1, 0, 1]),
+    sentiment_strength=st.floats(min_value=0.0, max_value=1.0),
+    impact=st.floats(min_value=0.0, max_value=1.0),
+    extraction_conf=st.floats(min_value=0.0, max_value=1.0),
+    source_cred=st.floats(min_value=0.0, max_value=1.0),
+    novelty=st.floats(min_value=0.0, max_value=1.0),
+    event_base_rate=st.floats(min_value=0.01, max_value=1.0),
+    ...
+)
+
+# Cluster LLR list generator
+cluster_llrs = st.lists(
+    st.floats(min_value=-2.5, max_value=2.5),
+    min_size=1, max_size=20,
+)
+
+# Regime generator
+regimes = st.sampled_from(["panic", "trend_following", "mean_reversion", "uncertainty"])
+```
+
+### Unit Test Coverage
+
+| Area | Key Example Tests |
+|---|---|
+| EvidenceUnit normalization | Company signal → correct fields; macro → correct horizon mapping |
+| q_i pipeline | Known inputs → known outputs for each sub-formula |
+| LLR conversion | p_correct=0.60 → LLR≈0.405; direction=-1 → negative LLR |
+| Clustering | 3 identical articles → n_eff < 3; independent → n_eff = N |
+| Posterior | Empty evidence → P_up=0.50; strong bullish → P_up > 0.60 |
+| Contradiction | All bullish → 0; equal split → high score |
+| Confidence | Zero data quality → near-zero confidence |
+| Macro LLR | Full exposure → max LLR ≈ 1.10; zero overlap → LLR ≈ 0 |
+| Competitive | Distance 4 → zero propagation; direct rival → attenuated signal |
+| Kelly sizing | p_win=0.3, b=2 → f_kelly < 0 → size = 0 |
+| Stops | Entry=100, stop_dist=0.02 → stop=98, TP > 100 |
+| Feature flag | Flag false → heuristic path; flag true → v3 path |
+| Error fallback | v3 raises → heuristic runs, error logged |
+
+### Integration Tests
+
+- Full pipeline: raw signals → EvidenceUnit → q_i → LLR → cluster → posterior → recommendation
+- Feature flag toggle: verify clean switch between pipelines mid-run
+- JSONB round-trip: store v3 output in PostgreSQL JSONB, retrieve and verify
+- Regime transitions: price series that crosses regime boundaries

.kiro/specs/math-core-v3-engine/requirements.md

@@ -0,0 +1,314 @@
+# Requirements Document
+
+## Introduction
+
+This specification defines the requirements for upgrading the Stonks Oracle signal processing engine from the current dual-mode pipeline (heuristic + probabilistic) to the v3 Calibrated Evidence Engine. The v3 engine replaces arbitrary weighted-sentiment scoring with a principled probabilistic pipeline: calibrated reliability estimation, log-likelihood ratio (LLR) evidence accumulation, correlation-aware clustering, Bayesian posterior assembly, return distribution modeling, and Kelly-criterion position sizing. The upgrade preserves the existing three-layer architecture (company, macro, competitive), the WeightedSignal abstraction, database schema compatibility, and the service boundary structure. The entire v3 engine operates behind a `v3_engine_enabled` feature flag with the heuristic mode retained as a fallback.
+
+## Glossary
+
+- **V3_Engine**: The new calibrated evidence engine that replaces the current heuristic and probabilistic scoring modes
+- **EvidenceUnit**: The canonical normalized shape for all signals (company, macro, competitive) before aggregation
+- **LLR**: Log-Likelihood Ratio — the calibrated evidence contribution of a single signal or cluster, measured in log-odds units
+- **Cluster**: A group of correlated signals sharing (symbol, horizon, event_type, source_group, time_bucket)
+- **n_eff**: Effective evidence count within a cluster after de-correlation adjustment
+- **Posterior**: The Bayesian posterior probability P_up computed via log-odds accumulation
+- **Regime**: Market regime classification (panic, trend_following, mean_reversion, uncertainty) derived from EMA trend and volatility indicators
+- **Contradiction_Score**: A measure of opposing evidence based on LLR entropy and evidence volume
+- **Data_Quality_Score**: Multiplicative fail-closed quality metric combining parse rate, confidence, freshness, coverage, and diversity
+- **Fractional_Kelly**: Position sizing method using Kelly criterion scaled by a conservative fraction (0.25) and further modulated by confidence, data quality, and contradiction
+- **Portfolio_Heat**: Total stop-defined risk dollars across all open positions as a fraction of portfolio value
+- **Feature_Flag**: The `v3_engine_enabled` runtime toggle that activates the v3 pipeline without code deployment
+- **Scoring_Service**: The `services/aggregation/scoring.py` module responsible for signal weight computation
+- **Bayesian_Service**: The `services/aggregation/bayesian.py` module responsible for posterior computation
+- **Contradiction_Service**: The `services/aggregation/contradiction.py` module responsible for conflict detection
+- **Regime_Service**: The `services/aggregation/regime.py` module responsible for market regime classification
+- **Projection_Service**: The `services/aggregation/projection.py` module responsible for trend projection
+- **Eligibility_Service**: The `services/recommendation/eligibility.py` module responsible for recommendation gating
+- **Position_Sizer**: The `services/trading/position_sizer.py` module responsible for trade sizing
+- **Stop_Loss_Manager**: The `services/trading/stop_loss_manager.py` module responsible for stop/TP computation
+- **Risk_Engine**: The `services/risk/engine.py` module responsible for portfolio risk enforcement
+
+## Requirements
+
+### Requirement 1: Canonical Evidence Unit Normalization
+
+**User Story:** As the aggregation engine, I want all signals normalized into a canonical EvidenceUnit shape, so that company, macro, and competitive signals flow through a single unified pipeline.
+
+#### Acceptance Criteria
+
+1. WHEN a company signal is received, THE V3_Engine SHALL normalize the signal into an EvidenceUnit containing symbol, layer (set to "company"), event_type, source_id, source_group, timestamp, horizon (one of: intraday, 1d, 7d, 30d, 90d), direction (-1/0/+1), sentiment_strength [0,1], impact [0,1], extraction_conf [0,1], source_cred [0,1], novelty [0,1], event_base_rate (float in (0.0, 1.0]), and cluster_id
+2. WHEN a macro signal is received, THE V3_Engine SHALL normalize the signal into an EvidenceUnit with layer set to "macro", symbol mapped from the macro impact record's ticker, direction mapped from impact_direction (positive→+1, negative→-1, neutral→0), impact mapped from macro_impact_score, source_cred mapped from event confidence, extraction_conf mapped from event confidence, novelty set to 1.0 for new events, source_id mapped from the global event id, source_group set to "macro", and horizon derived from the event's estimated_duration (short_term→7d, medium_term→30d, long_term→90d)
+3. WHEN a competitive signal is received, THE V3_Engine SHALL normalize the signal into an EvidenceUnit with layer set to "competitive", symbol set to the target ticker, direction mapped from signal_direction (bullish→+1, bearish→-1, neutral→0), impact mapped from signal_strength × relationship_strength, source_cred mapped from pattern_confidence, extraction_conf set to pattern_confidence, novelty set to 1.0, source_id mapped from source_document_id, source_group set to "competitive", and horizon derived from the pattern's time_horizon field
+4. THE V3_Engine SHALL assign direction value of +1 for signals with sentiment or impact_direction equal to "positive" or "bullish", -1 for "negative" or "bearish", and 0 for "neutral" or "mixed"
+5. WHEN a signal has direction equal to 0 (neutral), THE V3_Engine SHALL include the signal in quality, coverage, and contradiction context computations but SHALL exclude the signal from directional posterior voting
+6. IF a required source field (symbol, timestamp, or source_id) is missing or null in the incoming signal, THEN THE V3_Engine SHALL reject the signal, log a warning identifying the signal source and missing field, and not produce an EvidenceUnit for that signal
+7. IF an optional numeric field (sentiment_strength, impact, extraction_conf, source_cred, novelty) is missing or null, THEN THE V3_Engine SHALL substitute a default value of 0.5 for the missing field
+8. THE V3_Engine SHALL assign event_base_rate from a configured lookup by event_type, defaulting to 0.10 when no event_type-specific base rate is configured
+
+### Requirement 2: Calibrated Reliability Computation
+
+**User Story:** As the scoring engine, I want to compute a calibrated reliability q_i for each signal, so that evidence quality is measured probabilistically instead of via arbitrary weight products.
+
+#### Acceptance Criteria
+
+1. THE Scoring_Service SHALL compute extraction reliability as q_ext = sigmoid(k_ext × (extraction_conf - m_ext)) with defaults k_ext = 8.0 and m_ext = 0.55, where extraction_conf is in [0.0, 1.0] and q_ext output is in (0.0, 1.0)
+2. THE Scoring_Service SHALL compute source reliability using Bayesian shrinkage: E[theta_s] = (alpha_0 + hits_s) / (alpha_0 + beta_0 + hits_s + misses_s) with defaults alpha_0 = 3, beta_0 = 3, and q_source = clamp((E[theta_s] - 0.50) / 0.35, 0.0, 1.0), where hits_s and misses_s are non-negative integers representing the source's historical correct and incorrect directional predictions
+3. IF a source has zero historical outcomes (hits_s = 0 AND misses_s = 0), THEN THE Scoring_Service SHALL compute q_source = 0.0 from the prior (E[theta_s] = 0.5)
+4. THE Scoring_Service SHALL compute recency reliability as q_recency = 2^(-age_hours / tau_adaptive) where tau_adaptive = tau_base × (1 + 0.75 × impact + 0.50 × surprise) and surprise = clamp(-log2(event_base_rate) / 5, 0, 1), with age_hours = max((reference_time - signal_timestamp).total_seconds() / 3600, 0.0)
+5. IF event_base_rate is unavailable or equal to zero, THEN THE Scoring_Service SHALL use a default event_base_rate of 0.10 to prevent undefined logarithm computation
+6. THE Scoring_Service SHALL use horizon-specific half-life defaults: intraday=2h, 1d=12h, 7d=72h, 30d=240h, 90d=720h
+7. THE Scoring_Service SHALL compute uniqueness as q_uniqueness = clamp(0.50 + 0.50 × novelty, 0.50, 1.00) × (1 / sqrt(1 + duplicate_count_before)), where duplicate_count_before is the number of other signals in the same cluster that were ingested before this signal
+8. THE Scoring_Service SHALL compute final signal reliability as q_i = clamp(q_ext × q_source × source_cred × q_recency × q_uniqueness, 0.0, 1.0)
+9. WHEN q_recency falls below 0.01, THE Scoring_Service SHALL apply a floor of 0.01 only for explainability display output and SHALL use the unmodified q_recency value (including zero) for posterior voting computation
+
+### Requirement 3: Log-Likelihood Ratio Conversion
+
+**User Story:** As the posterior engine, I want signals converted to calibrated log-likelihood ratios, so that evidence accumulation follows proper Bayesian updating rules.
+
+#### Acceptance Criteria
+
+1. THE Scoring_Service SHALL compute directional correctness probability as p_correct = clamp(0.50 + 0.35 × q_i × impact × sentiment_strength, 0.501, 0.85)
+2. THE Scoring_Service SHALL compute the signal log-likelihood ratio as LLR_i = direction × ln(p_correct / (1 - p_correct)), where ln denotes the natural logarithm (base e), consistent with the logit function used in posterior assembly
+3. WHEN direction equals 0 (neutral signal), THE Scoring_Service SHALL produce LLR_i = 0.0, excluding the signal from directional posterior voting while retaining it for quality and contradiction context
+4. THE Scoring_Service SHALL clamp p_correct to a minimum of 0.501 to ensure LLR_i is always nonzero for directional signals (direction ≠ 0), producing a minimum absolute LLR magnitude of approximately 0.004
+5. THE Scoring_Service SHALL clamp p_correct to a maximum of 0.85 to prevent any single signal from dominating the posterior, producing a maximum absolute LLR magnitude of approximately 1.735
+6. FOR ALL valid directional signals (direction ∈ {-1, +1}), THE Scoring_Service SHALL produce LLR_i values with the same sign as direction
+
+### Requirement 4: Correlation-Aware Evidence Clustering
+
+**User Story:** As the aggregation engine, I want correlated signals grouped and de-duplicated before posterior assembly, so that near-identical articles cannot inflate evidence counts.
+
+#### Acceptance Criteria
+
+1. THE V3_Engine SHALL cluster signals by (symbol, horizon, event_type, source_group, time_bucket)
+2. THE V3_Engine SHALL compute effective evidence count as n_eff_c = (sum_i w_i)^2 / (sum_i w_i^2 + 2 × sum_{i<j}(rho_ij × w_i × w_j)) where w_i = abs(LLR_i)
+3. THE V3_Engine SHALL use default pairwise correlations: rho=0.80 for same wire/story/source group, rho=0.50 for same event different publisher, rho=0.25 for same theme different event, rho=0.00 for independent events
+4. THE V3_Engine SHALL compute cluster LLR as LLR_c = weighted_mean(LLR_i, abs(LLR_i)) × sqrt(n_eff_c)
+5. THE V3_Engine SHALL clamp each cluster LLR to the range [-2.5, 2.5] to prevent any single cluster from dominating the posterior
+
+### Requirement 5: Posterior Assembly via Log-Odds
+
+**User Story:** As the Bayesian engine, I want to assemble a posterior probability from cluster LLRs and a regime-aware prior, so that the trading decision is based on calibrated belief.
+
+#### Acceptance Criteria
+
+1. THE Bayesian_Service SHALL use a neutral base prior of P_prior = 0.50 unless a calibrated symbol/sector prior is stored in the risk_configs table for the given ticker or its sector
+2. THE Bayesian_Service SHALL compute posterior log-odds as logit(P_up) = logit(P_prior) + sum_c(gamma_regime × LLR_c) where gamma_regime is the regime evidence multiplier from Requirement 6 criterion 6
+3. THE Bayesian_Service SHALL compute P_up = sigmoid(logit(P_up)) = 1/(1+exp(-logit(P_up))) and P_down = 1 - P_up, clamping P_up to the range [1e-10, 1 - 1e-10] to avoid numerical boundary issues
+4. THE Bayesian_Service SHALL compute trend strength as strength = abs(2 × P_up - 1), producing a value in [0.0, 1.0] where 0.0 indicates maximum uncertainty and 1.0 indicates maximum directional conviction
+5. THE Bayesian_Service SHALL apply regime-specific direction thresholds to classify direction from P_up: panic (bullish when P_up >= 0.68, bearish when P_up <= 0.32), trend_following (bullish when P_up >= 0.60, bearish when P_up <= 0.40), mean_reversion (bullish when P_up >= 0.63, bearish when P_up <= 0.37), uncertainty (bullish when P_up >= 0.65, bearish when P_up <= 0.35). WHEN P_up falls between the bullish and bearish thresholds, THE Bayesian_Service SHALL classify direction as neutral
+6. WHEN P_prior is calibrated from the risk_configs table, THE Bayesian_Service SHALL clamp P_prior to the range [0.40, 0.60] before computing logit(P_prior)
+7. IF the risk_configs lookup for a calibrated prior fails due to a database error, THEN THE Bayesian_Service SHALL fall back to the neutral base prior of 0.50 and log a warning
+
+### Requirement 6: Regime Detection v3
+
+**User Story:** As the regime service, I want to classify market regimes using z-scored indicators and apply regime-appropriate evidence multipliers, so that the engine adapts its sensitivity to market conditions.
+
+#### Acceptance Criteria
+
+1. THE Regime_Service SHALL compute trend_z = (EMA_20 - EMA_100) / ATR_20 where EMA_20 and EMA_100 are exponential moving averages of closing prices, and ATR_20 is the 20-day Average True Range. THE Regime_Service SHALL compute vol_ratio = sigma_20 / sigma_100 where sigma_20 and sigma_100 are standard deviations of daily returns
+2. THE Regime_Service SHALL classify panic when vol_ratio > 1.5 OR abs(trend_z) > 2.5. Panic classification SHALL take priority over all other regimes
+3. IF the regime is not panic, THEN THE Regime_Service SHALL classify trend_following when abs(trend_z) >= 0.75 AND vol_ratio < 1.3
+4. IF the regime is neither panic nor trend_following, THEN THE Regime_Service SHALL classify mean_reversion when abs(trend_z) < 0.50 AND vol_ratio < 1.0
+5. THE Regime_Service SHALL classify uncertainty for all conditions not matching panic, trend_following, or mean_reversion
+6. THE Regime_Service SHALL apply regime evidence multipliers (gamma_regime): panic=0.70, trend_following=1.10, mean_reversion=0.90, uncertainty=0.80
+7. THE Regime_Service SHALL apply regime confidence multipliers: panic=0.70, trend_following=1.00, mean_reversion=0.95, uncertainty=0.85
+8. IF market data is insufficient to compute EMA_100 (fewer than 100 closing prices) or ATR_20 (fewer than 20 bars) or sigma_100 (fewer than 100 daily returns), THEN THE Regime_Service SHALL default to the uncertainty regime
+
+### Requirement 7: LLR Entropy Contradiction
+
+**User Story:** As the contradiction service, I want to measure meaningful opposing evidence using LLR entropy, so that contradiction reflects genuine disagreement scaled by evidence volume.
+
+#### Acceptance Criteria
+
+1. THE Contradiction_Service SHALL compute E_pos = sum_c(max(LLR_c, 0)) and E_neg = sum_c(max(-LLR_c, 0)) and E_total = E_pos + E_neg
+2. WHEN E_total equals zero (no directional evidence from any cluster), THE Contradiction_Service SHALL return a contradiction score of 0.0
+3. WHEN E_total is greater than zero, THE Contradiction_Service SHALL compute f_pos = E_pos / E_total and f_neg = E_neg / E_total where f_pos + f_neg = 1.0
+4. THE Contradiction_Service SHALL compute H_conflict = -f_pos × log2(f_pos) - f_neg × log2(f_neg), treating 0 × log2(0) as 0 for the boundary case. H_conflict ranges from 0.0 (all one direction) to 1.0 (equal split)
+5. THE Contradiction_Service SHALL compute volume_factor = 1 - exp(-E_total / 3.0), where 3.0 represents the evidence mass at which contradiction becomes 95% significant
+6. THE Contradiction_Service SHALL compute the final contradiction score as H_conflict × volume_factor, producing a value in [0.0, 1.0]
+7. WHEN only one direction of evidence exists (E_pos = 0 or E_neg = 0 but E_total > 0), THE Contradiction_Service SHALL return a contradiction score of 0.0
+
+### Requirement 8: Multiplicative Confidence v3
+
+**User Story:** As the aggregation engine, I want confidence computed multiplicatively from independent quality dimensions, so that one weak dimension suppresses the trade rather than being averaged away.
+
+#### Acceptance Criteria
+
+1. THE V3_Engine SHALL compute C_evidence = 1 - exp(-n_eff_total / 5.0) where n_eff_total = sum_c(n_eff_c)
+2. THE V3_Engine SHALL compute C_quality = weighted_mean(q_i, weight=abs(LLR_i))
+3. THE V3_Engine SHALL compute confidence = clamp(C_evidence × sqrt(C_quality) × sqrt(max(strength, 0.05)) × regime_confidence_multiplier × (1 - contradiction) × data_quality_score, 0, 1)
+4. THE V3_Engine SHALL use strength = abs(2 × P_up - 1) as the directional separation term
+5. WHEN any single confidence dimension is near zero, THE V3_Engine SHALL produce a near-zero final confidence due to the multiplicative formula
+
+### Requirement 9: Macro Layer v3 (Noisy-OR Exposure)
+
+**User Story:** As the macro interpolation service, I want to compute normalized exposure via noisy-OR and emit macro evidence as LLR into the shared posterior, so that macro signals integrate with company evidence without special post-hoc modifiers.
+
+#### Acceptance Criteria
+
+1. THE V3_Engine SHALL compute macro exposure as E_raw = 1 - product_k(1 - w_k × O_k) with default weights w_geo=0.35, w_supply=0.25, w_commodity=0.25, w_sector=0.15
+2. THE V3_Engine SHALL normalize macro exposure as E_macro = E_raw / E_max where E_max = 1 - product_k(1 - w_k)
+3. THE V3_Engine SHALL apply resilience dampener per market position tier: global_leader=0.70, multinational=0.85, regional=1.00, domestic=1.20
+4. THE V3_Engine SHALL compute macro likelihood ratio as LLR_macro = macro_direction × log(p_macro / (1 - p_macro)) where p_macro = clamp(0.50 + 0.30 × macro_impact × event_confidence × q_recency, 0.501, 0.80)
+5. THE V3_Engine SHALL feed macro LLR into the same posterior engine as company evidence without requiring a separate post-hoc modifier
+
+### Requirement 10: Competitive Layer v3 (Correlation-Shrunk Propagation)
+
+**User Story:** As the signal propagation service, I want to use correlation-shrunk attenuation for competitive signals, so that propagated evidence is properly discounted by distance and relationship strength.
+
+#### Acceptance Criteria
+
+1. THE V3_Engine SHALL compute shrunk correlation as rho_shrunk = (n / (n + 30)) × rho_rolling + (30 / (n + 30)) × rho_prior with rho_prior_same_sector = 0.30 and rho_prior_cross_sector = 0.10
+2. THE V3_Engine SHALL compute rho_effective = max(rho_shrunk, 0) to use only positive propagation unless the relationship is explicitly inverse
+3. THE V3_Engine SHALL compute graph attenuation as attenuation = rho_effective × exp(-0.85 × d_network) with max_distance = 3
+4. THE V3_Engine SHALL compute competitive LLR as LLR_competitive = LLR_source × attenuation × pattern_confidence
+5. THE V3_Engine SHALL clamp competitive LLR to the range [-1.25, 1.25] to prevent competitive signals from dominating the posterior
+
+### Requirement 11: Trend Projection v3 (Posterior State)
+
+**User Story:** As the projection service, I want to project trends using a posterior state with regime-aware decay, so that projections are grounded in the same Bayesian framework as current estimates.
+
+#### Acceptance Criteria
+
+1. THE Projection_Service SHALL maintain an evidence state A_t = phi_regime × A_{t-1} + sum_c(LLR_c), initialized to A_0 = 0.0 when no prior state exists for a ticker-horizon pair
+2. THE Projection_Service SHALL use regime-specific decay factors: panic phi=0.35, trend_following phi=0.80, mean_reversion phi=0.55, uncertainty phi=0.50
+3. THE Projection_Service SHALL compute projected alpha as A_projected_h = phi_regime^h × A_t + expected_known_catalyst_LLR_h, where h is the projection horizon in aggregation cycles and expected_known_catalyst_LLR_h defaults to 0.0 when no known catalysts exist
+4. THE Projection_Service SHALL compute projected probability as P_up_projected_h = sigmoid(logit(P_prior_h) + A_projected_h)
+5. THE Projection_Service SHALL compute projected strength as abs(2 × P_up_projected_h - 1)
+6. THE Projection_Service SHALL flag divergence when sign(P_up_projected_h - 0.5) differs from sign(P_up_t - 0.5)
+7. WHEN market data is insufficient for regime classification, THE Projection_Service SHALL use the uncertainty decay factor (phi=0.50) as the default
+
+### Requirement 12: Return Distribution and Expected Value Gate
+
+**User Story:** As the eligibility service, I want to gate recommendations using a return distribution model, so that only trades with positive risk-adjusted expected value pass through.
+
+#### Acceptance Criteria
+
+1. THE Eligibility_Service SHALL compute horizon volatility as sigma_h = realized_vol_20d × sqrt(horizon_days / 252), where horizon_days maps to 1 (intraday/1d), 7 (7d), 30 (30d), or 90 (90d)
+2. THE Eligibility_Service SHALL compute expected return as mu_h = tanh(A_projected_h / 3.0) × confidence × sigma_h
+3. THE Eligibility_Service SHALL compute EV_long = mu_h - costs - 0.10 × CVaR_5_loss, where costs = spread_cost + slippage_estimate + commission_estimate, and CVaR_5_loss = sigma_h × 1.645 × 1.4 (Gaussian approximation of expected loss beyond the 5th percentile)
+4. THE Eligibility_Service SHALL compute regime-specific minimum edge: panic=0.0100, trend_following=0.0035, mean_reversion=0.0050, uncertainty=0.0075
+5. THE Eligibility_Service SHALL require EV_long > min_edge AND EV_long > max(0.0025, 0.25 × costs) for trade eligibility
+6. THE Eligibility_Service SHALL require confidence >= regime_confidence_min AND contradiction <= regime_contradiction_max AND n_eff_total >= 2.0 AND data_quality_score >= 0.50 for eligibility
+7. IF realized_vol_20d is unavailable (fewer than 20 trading days of price data), THEN THE Eligibility_Service SHALL use a default volatility of 0.25 (annualized) for sigma_h computation
+
+### Requirement 13: Recommendation Eligibility and Mode Escalation v3
+
+**User Story:** As the recommendation service, I want regime-aware eligibility gates and mode escalation, so that recommendation quality matches the rigor of the v3 posterior.
+
+#### Acceptance Criteria
+
+1. THE Eligibility_Service SHALL apply regime-specific eligibility thresholds: panic (confidence_min=0.70, contradiction_max=0.25, strength_min=0.36), trend_following (confidence_min=0.55, contradiction_max=0.40, strength_min=0.20), mean_reversion (confidence_min=0.60, contradiction_max=0.35, strength_min=0.26), uncertainty (confidence_min=0.65, contradiction_max=0.30, strength_min=0.30)
+2. THE Eligibility_Service SHALL map action as BUY when P_up >= bullish_threshold and EV_long > min_edge, SELL when existing position and EV_exit > EV_hold, HOLD when existing position, and WATCH otherwise
+3. THE Eligibility_Service SHALL escalate to live_eligible when action is BUY or SELL and confidence >= 0.75 and contradiction <= 0.20 and n_eff_total >= 5 and EV_long > 2 × min_edge and risk_engine_passed
+4. THE Eligibility_Service SHALL escalate to paper_eligible when action is BUY or SELL and confidence >= 0.60 and EV_long > min_edge and risk_engine_passed
+5. IF eligibility gates are not met, THEN THE Eligibility_Service SHALL assign mode as informational
+
+### Requirement 14: Fractional Kelly Position Sizing
+
+**User Story:** As the position sizer, I want to size positions using fractional Kelly criterion, so that position sizes are proportional to edge and constrained by risk.
+
+#### Acceptance Criteria
+
+1. THE Position_Sizer SHALL compute stop_distance_pct = max(ATR_pct × ATR_multiplier_regime, sigma_h × 1.25, 0.005) where ATR_pct = ATR_14 / current_price, with regime ATR multipliers: panic=2.5, trend_following=1.8, mean_reversion=1.4, uncertainty=2.0
+2. THE Position_Sizer SHALL compute reward ratio b = clamp(1.2 + 2.0 × confidence + 1.0 × strength - contradiction, 1.2, 3.0)
+3. THE Position_Sizer SHALL compute f_kelly = (p_win × b - (1 - p_win)) / b where p_win = P_up from the Bayesian posterior
+4. THE Position_Sizer SHALL compute final sizing as portfolio_pct = clamp(max(0, f_kelly) × 0.25 × confidence × data_quality_score × (1 - contradiction), 0, max_position_pct)
+5. THE Position_Sizer SHALL enforce hard caps by reducing portfolio_pct to the minimum of: max_position_pct from the active risk tier, available_sector_capacity_pct, available_correlation_capacity_pct (0 if weighted average absolute correlation with existing positions exceeds 0.80), and available_heat_capacity_pct
+6. IF portfolio_pct after all caps is less than 0.005 (0.5% of portfolio), THEN THE Position_Sizer SHALL downgrade the recommendation to WATCH with reason "position_below_minimum"
+7. IF f_kelly is less than or equal to zero, THEN THE Position_Sizer SHALL produce portfolio_pct = 0 and downgrade the recommendation to WATCH with reason "negative_edge"
+
+### Requirement 15: Stop-Defined Portfolio Heat
+
+**User Story:** As the risk engine, I want portfolio heat calculated from stop-defined risk dollars, so that risk measurement reflects actual loss exposure rather than position notional.
+
+#### Acceptance Criteria
+
+1. THE Risk_Engine SHALL compute risk_dollars = position_value × stop_distance_pct for each open position
+2. THE Risk_Engine SHALL compute portfolio_heat = sum of risk_dollars across all open positions
+3. IF portfolio_heat exceeds max_portfolio_heat × portfolio_value, THEN THE Risk_Engine SHALL reject new position entries
+4. THE Position_Sizer SHALL compute available_heat_capacity = max_portfolio_heat × portfolio_value - current_portfolio_heat
+5. THE Position_Sizer SHALL reject a position when the new risk_dollars would exceed available_heat_capacity
+
+### Requirement 16: Regime-Aware Stop Loss and Take Profit
+
+**User Story:** As the stop loss manager, I want stops and targets computed from regime-aware volatility and dynamic reward ratios, so that exit levels adapt to current market conditions.
+
+#### Acceptance Criteria
+
+1. THE Stop_Loss_Manager SHALL compute stop_distance_pct = max(ATR_pct × regime_ATR_multiplier, sigma_h × z_stop, min_stop_pct) with z_stop = 1.25 and min_stop_pct = 0.005
+2. THE Stop_Loss_Manager SHALL compute stop_loss = entry_price × (1 - stop_distance_pct) for long positions
+3. THE Stop_Loss_Manager SHALL compute take_profit = entry_price × (1 + b × stop_distance_pct) where b = clamp(1.2 + 2.0 × confidence + 1.0 × strength - contradiction, 1.2, 3.0)
+4. THE Stop_Loss_Manager SHALL activate trailing stop when unrealized_gain_pct >= 0.50 × take_profit_distance_pct
+5. THE Stop_Loss_Manager SHALL compute trailing_stop = max(existing_stop, current_price × (1 - trailing_distance_pct)) where trailing_distance_pct = max(ATR_pct × trailing_ATR_mult, sigma_h × 0.75)
+
+### Requirement 17: Data Quality v3 (Multiplicative Fail-Closed)
+
+**User Story:** As the suppression layer, I want data quality computed as a multiplicative fail-closed metric, so that a single catastrophic quality failure suppresses the entire recommendation.
+
+#### Acceptance Criteria
+
+1. THE V3_Engine SHALL compute Q_parse = 1 - extraction_failure_rate
+2. THE V3_Engine SHALL compute Q_conf = weighted_mean(extraction_conf_i, weight=impact_i)
+3. THE V3_Engine SHALL compute Q_fresh = exp(-age_newest_hours / 168)
+4. THE V3_Engine SHALL compute Q_coverage = 1 - exp(-N_valid / 5)
+5. THE V3_Engine SHALL compute Q_diversity = min(1, log2(1 + N_source_types) / log2(4))
+6. THE V3_Engine SHALL compute data_quality_score = clamp(Q_parse × sqrt(Q_conf) × Q_fresh × Q_coverage × Q_diversity, 0, 1)
+7. IF data_quality_score < 0.50 OR N_valid < 2 OR Q_parse < 0.50, THEN THE V3_Engine SHALL force the recommendation to informational mode
+8. IF company evidence is zero and only macro or competitive evidence exists, THEN THE V3_Engine SHALL force the recommendation to informational mode unless macro_only_enabled is configured
+
+### Requirement 18: Risk Tier Auto-Adjustment v3
+
+**User Story:** As the risk tier controller, I want tier adjustments based on risk-adjusted performance metrics, so that the engine self-corrects when performance degrades.
+
+#### Acceptance Criteria
+
+1. THE Risk_Engine SHALL track profit_factor_30d (gross_profit / gross_loss over 30 days), max_drawdown_30d (largest peak-to-trough portfolio decline over 30 days as a fraction), calibration_error (mean absolute difference between predicted P_up and realized binary outcome over 30 days), and realized_sharpe_30d (annualized Sharpe ratio of daily returns over 30 days)
+2. THE Risk_Engine SHALL evaluate tier adjustment conditions once per calendar day after the trading session closes
+3. THE Risk_Engine SHALL downgrade one tier if any condition is met: profit_factor_30d < 1.0 OR max_drawdown_30d > 0.12 OR calibration_error > 0.20 OR realized_sharpe_30d < 0
+4. THE Risk_Engine SHALL upgrade one tier only if all conditions are met: profit_factor_30d > 1.35 AND max_drawdown_30d < 0.05 AND calibration_error < 0.12 AND reserve_pool > 0.20 AND N_trades_30d >= 20
+5. IF a downgrade condition is triggered, THEN THE Risk_Engine SHALL apply the downgrade immediately without waiting for an upgrade evaluation
+6. THE Risk_Engine SHALL enforce a minimum cooldown of 7 calendar days between consecutive upgrade evaluations to prevent tier oscillation
+
+### Requirement 19: Feature Flag and Fallback
+
+**User Story:** As an operator, I want the v3 engine gated behind a runtime feature flag with the heuristic mode as fallback, so that the upgrade can be rolled out safely without downtime.
+
+#### Acceptance Criteria
+
+1. WHILE `v3_engine_enabled` is False, THE V3_Engine SHALL route all aggregation through the existing heuristic pipeline without any v3 computation
+2. WHILE `v3_engine_enabled` is True, THE V3_Engine SHALL route all aggregation through the v3 calibrated evidence pipeline
+3. THE V3_Engine SHALL read the `v3_engine_enabled` flag from the risk_configs table at the start of each aggregation cycle without requiring a service restart
+4. IF the v3 pipeline encounters an unhandled error during aggregation, THEN THE V3_Engine SHALL log the error at ERROR level with full traceback and fall back to heuristic mode for that aggregation cycle, recording the fallback event in output metadata
+5. THE V3_Engine SHALL store a `pipeline_mode` field value of "v3" or "heuristic" in all output records (TrendSummary, Recommendation) JSONB metadata indicating which pipeline produced the result
+6. IF the `v3_engine_enabled` flag cannot be read from the database (connection error or missing row), THEN THE V3_Engine SHALL default to heuristic mode and log a warning
+
+### Requirement 20: Database Compatibility and Output Contract
+
+**User Story:** As the system architect, I want v3 output stored in existing tables using JSONB metadata, so that no schema migration or downtime is required.
+
+#### Acceptance Criteria
+
+1. THE V3_Engine SHALL store posterior fields (p_up, log_odds, strength, confidence, contradiction, n_eff, data_quality) in the existing TrendSummary JSONB metadata column
+2. THE V3_Engine SHALL store return model fields (mu_h, sigma_h, ev_long, min_edge) in the Recommendation JSONB metadata column
+3. THE V3_Engine SHALL expose an explainability payload containing top_positive_clusters, top_negative_clusters, suppression_reasons, and risk_adjustments
+4. THE V3_Engine SHALL preserve the existing WeightedSignal abstraction as an intermediate representation before LLR conversion
+5. THE V3_Engine SHALL preserve all existing database table schemas without requiring new migrations for core functionality
+
+### Requirement 21: Mathematical Correctness Properties
+
+**User Story:** As a developer, I want property-based tests validating all v3 mathematical invariants, so that correctness is verified across the input space.
+
+#### Acceptance Criteria
+
+1. FOR ALL valid EvidenceUnits, THE V3_Engine SHALL produce q_i values in the range [0, 1]
+2. FOR ALL valid q_i values, THE V3_Engine SHALL produce p_correct values in the range [0.501, 0.85]
+3. FOR ALL valid signals with direction != 0, THE V3_Engine SHALL produce LLR values with the same sign as direction
+4. FOR ALL valid cluster configurations, THE V3_Engine SHALL produce n_eff_c values satisfying 0 < n_eff_c <= N (where N is the cluster size)
+5. FOR ALL valid cluster LLRs, THE Bayesian_Service SHALL produce P_up values in the range (0, 1) exclusive
+6. FOR ALL valid inputs, THE V3_Engine SHALL produce confidence values in the range [0, 1]
+7. FOR ALL valid inputs with no opposing evidence, THE Contradiction_Service SHALL produce a contradiction score of 0
+8. FOR ALL valid inputs, THE Position_Sizer SHALL produce portfolio_pct values in the range [0, max_position_pct]
+9. FOR ALL valid inputs where f_kelly <= 0, THE Position_Sizer SHALL produce a portfolio_pct of 0
+10. FOR ALL valid EvidenceUnit sequences, serializing the posterior state to JSON and deserializing SHALL produce an equivalent posterior state (round-trip property)

.kiro/specs/math-core-v3-engine/tasks.md

@@ -0,0 +1,451 @@
+# Implementation Plan: Math Core v3 Engine
+
+## Overview
+
+Incremental upgrade of the Stonks Oracle signal processing engine from dual-mode heuristic/probabilistic to the v3 Calibrated Evidence Engine. Implementation follows the v3 math doc section 20 order: EvidenceUnit → reliability → LLR → clustering → posterior → contradiction → confidence → macro/competitive layers → EV gate → Kelly sizing → stop-defined heat → retire heuristic scoring. All v3 code operates behind the `v3_engine_enabled` feature flag with heuristic fallback.
+
+## Tasks
+
+- [x] 1. EvidenceUnit and LLR conversion behind feature flag
+  - [x] 1.1 Implement EvidenceUnit dataclass and normalization functions in `services/aggregation/scoring.py`
+    - Add frozen dataclass `EvidenceUnit` with all 16 fields (symbol, layer, event_type, source_id, source_group, timestamp, horizon, direction, sentiment_strength, impact, extraction_conf, source_cred, novelty, event_base_rate, cluster_id)
+    - Implement `normalize_company_signal()`, `normalize_macro_signal()`, `normalize_competitive_signal()`
+    - Validate required fields (symbol, timestamp, source_id) — reject with warning on missing
+    - Substitute 0.5 for missing optional numeric fields
+    - Map direction from sentiment/impact_direction strings (+1/-1/0)
+    - Assign event_base_rate from EVENT_TYPE_BASE_RATES lookup (default 0.10)
+    - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8_
+
+  - [x] 1.2 Implement calibrated reliability pipeline (`compute_v3_reliability`) in `services/aggregation/scoring.py`
+    - Add `SourceStats` dataclass (source_id, hits, misses, alpha_0=3, beta_0=3)
+    - Add `ReliabilityComponents` dataclass (q_ext, q_source, q_recency, q_uniqueness, q_i)
+    - Implement q_ext = sigmoid(8.0 × (extraction_conf - 0.55))
+    - Implement q_source via Bayesian shrinkage: E[theta_s] = (alpha_0 + hits) / (alpha_0 + beta_0 + hits + misses), then clamp((E - 0.50) / 0.35, 0, 1)
+    - Implement q_recency = 2^(-age_hours / tau_adaptive) with adaptive half-life formula
+    - Implement q_uniqueness = clamp(0.5 + 0.5 × novelty, 0.5, 1.0) × (1 / sqrt(1 + dup_count))
+    - Implement q_i = clamp(q_ext × q_source × source_cred × q_recency × q_uniqueness, 0, 1)
+    - Apply floor of 0.01 on q_recency only for explainability display
+    - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9_
+
+  - [x] 1.3 Implement LLR conversion (`compute_llr`) in `services/aggregation/scoring.py`
+    - Compute p_correct = clamp(0.50 + 0.35 × q_i × impact × sentiment_strength, 0.501, 0.85)
+    - Compute LLR_i = direction × ln(p_correct / (1 - p_correct))
+    - Return 0.0 for neutral signals (direction == 0)
+    - Ensure LLR sign always matches direction for directional signals
+    - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_
+
+  - [x] 1.4 Add feature flag routing in `services/aggregation/worker.py`
+    - Read `v3_engine_enabled` from risk_configs table at start of each aggregation cycle
+    - Route to v3 pipeline when True, heuristic when False
+    - Default to heuristic mode if DB read fails (log warning)
+    - Wrap v3 pipeline in try/except — fall back to heuristic on unhandled error (log ERROR with traceback)
+    - Store `pipeline_mode` field ("v3" or "heuristic") in output metadata
+    - _Requirements: 19.1, 19.2, 19.3, 19.4, 19.5, 19.6_
+
+  - [x] 1.5 Write property tests for EvidenceUnit, reliability, and LLR (`tests/test_pbt_v3_reliability.py`)
+    - **Property 1: Reliability q_i is bounded in [0, 1]**
+    - **Property 2: p_correct is bounded in [0.501, 0.85]**
+    - **Property 3: LLR sign matches direction and magnitude is bounded**
+    - **Property 4: Neutral signals produce zero LLR**
+    - **Property 21: EvidenceUnit normalization preserves field ranges**
+    - **Validates: Requirements 1.1–1.8, 2.1–2.9, 3.1–3.6, 21.1–21.3**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 1.6 Write unit tests for EvidenceUnit normalization and LLR conversion (`tests/test_v3_evidence_unit.py`)
+    - Test company signal → EvidenceUnit with correct field mapping
+    - Test macro signal → EvidenceUnit with correct horizon mapping (short_term→7d, medium_term→30d, long_term→90d)
+    - Test competitive signal → EvidenceUnit with correct direction mapping
+    - Test missing required fields → rejection with warning
+    - Test missing optional fields → default 0.5 substitution
+    - Test known inputs through full q_i pipeline → expected outputs
+    - Test LLR boundary cases: p_correct at clamp boundaries
+    - _Requirements: 1.1–1.8, 2.1–2.9, 3.1–3.6_
+
+- [x] 2. Evidence clustering and n_eff
+
+  - [x] 2.1 Implement correlation-aware clustering in `services/aggregation/worker.py`
+    - Add `EvidenceCluster` dataclass (cluster_id, units, llrs, n_eff, cluster_llr)
+    - Implement `cluster_evidence()` — group EvidenceUnits by (symbol, horizon, event_type, source_group, time_bucket)
+    - Compute cluster_id as hash of grouping key
+    - Define time_bucket resolution per horizon (intraday=1h, 1d=4h, 7d=24h, 30d=72h, 90d=168h)
+    - _Requirements: 4.1_
+
+  - [x] 2.2 Implement n_eff computation in `services/aggregation/worker.py`
+    - Implement `compute_n_eff(llrs, correlations)` using formula: (sum w_i)² / (sum w_i² + 2 × sum_{i<j} rho_ij × w_i × w_j)
+    - Use default pairwise correlations: same wire=0.80, same event diff publisher=0.50, same theme diff event=0.25, independent=0.00
+    - Guard against division by zero (denominator → return n_eff=1.0)
+    - _Requirements: 4.2, 4.3_
+
+  - [x] 2.3 Implement cluster LLR computation in `services/aggregation/worker.py`
+    - Implement `compute_cluster_llr(llrs, n_eff)` = clamp(weighted_mean(LLR_i, |LLR_i|) × sqrt(n_eff), -2.5, 2.5)
+    - Handle single-signal clusters (LLR_c = LLR_i clamped)
+    - Handle all-zero LLR clusters (cluster_llr = 0.0)
+    - _Requirements: 4.4, 4.5_
+
+  - [x] 2.4 Write property tests for clustering (`tests/test_pbt_v3_clustering.py`)
+    - **Property 5: Effective evidence count n_eff is bounded by cluster size**
+    - **Property 6: Cluster LLR is clamped to [-2.5, 2.5]**
+    - **Validates: Requirements 4.2, 4.3, 4.4, 4.5, 21.4**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 2.5 Write unit tests for clustering (`tests/test_v3_clustering.py`)
+    - Test 3 identical articles from same source → n_eff < 3
+    - Test 3 independent articles → n_eff ≈ 3
+    - Test single signal cluster → n_eff = 1.0
+    - Test cluster LLR clamp at ±2.5
+    - Test grouping by correct key dimensions
+    - _Requirements: 4.1–4.5_
+
+- [x] 3. Checkpoint - Verify foundation layer
+  - Ensure all tests pass for EvidenceUnit, reliability, LLR, and clustering.
+  - Ensure all tests pass, ask the user if questions arise.
+
+- [x] 4. Replace trend assembly with posterior P_up
+
+  - [x] 4.1 Implement regime detection v3 in `services/aggregation/regime.py`
+    - Add `V3RegimeClassification` dataclass (regime, trend_z, vol_ratio, evidence_multiplier, confidence_multiplier, phi_decay, atr_multiplier)
+    - Implement `classify_regime_v3(closing_prices, daily_returns, atr_20)` using ATR-normalized trend_z = (EMA_20 - EMA_100) / ATR_20
+    - Compute vol_ratio = sigma_20 / sigma_100
+    - Classification rules in priority: panic (vol_ratio > 1.5 OR |trend_z| > 2.5), trend_following (|trend_z| >= 0.75 AND vol_ratio < 1.3), mean_reversion (|trend_z| < 0.50 AND vol_ratio < 1.0), uncertainty (default)
+    - Assign regime parameters: gamma, confidence_mult, phi, ATR_mult, min_edge
+    - Default to uncertainty when data insufficient
+    - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8_
+
+  - [x] 4.2 Implement posterior assembly via log-odds in `services/aggregation/bayesian.py`
+    - Add `V3Posterior` dataclass (p_up, p_down, log_odds, strength, direction, n_eff_total, regime)
+    - Implement `compute_v3_posterior(clusters, regime, p_prior=0.50)`
+    - Compute logit(P_up) = logit(P_prior) + sum(gamma_regime × LLR_c)
+    - Compute P_up = sigmoid(log_odds), clamp to [1e-10, 1 - 1e-10]
+    - Compute strength = abs(2 × P_up - 1)
+    - Classify direction using regime-specific thresholds (panic: 0.68/0.32, trend_following: 0.60/0.40, mean_reversion: 0.63/0.37, uncertainty: 0.65/0.35)
+    - Load calibrated prior from risk_configs if available (clamp to [0.40, 0.60]), fall back to 0.50 on error
+    - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7_
+
+  - [x] 4.3 Write property tests for posterior and regime (`tests/test_pbt_v3_posterior.py`)
+    - **Property 7: Posterior P_up is in open interval (0, 1)**
+    - **Property 20: Regime classification is exhaustive and deterministic**
+    - **Validates: Requirements 5.3, 6.2–6.5, 21.5**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 4.4 Write unit tests for posterior assembly (`tests/test_v3_posterior.py`)
+    - Test empty clusters → P_up = 0.50 (neutral prior)
+    - Test all bullish clusters → P_up > 0.50
+    - Test regime direction thresholds at boundary values
+    - Test prior clamp [0.40, 0.60]
+    - Test regime classification with known inputs
+    - _Requirements: 5.1–5.7, 6.1–6.8_
+
+- [x] 5. Replace contradiction with LLR entropy
+
+  - [x] 5.1 Implement LLR entropy contradiction in `services/aggregation/contradiction.py`
+    - Add `compute_v3_contradiction(clusters: list[EvidenceCluster]) -> float`
+    - Compute E_pos = sum(max(LLR_c, 0)), E_neg = sum(max(-LLR_c, 0)), E_total = E_pos + E_neg
+    - When E_total == 0 → return 0.0
+    - When only one direction exists (E_pos == 0 or E_neg == 0) → return 0.0
+    - Compute f_pos = E_pos / E_total, f_neg = E_neg / E_total
+    - Compute H_conflict = -f_pos × log2(f_pos) - f_neg × log2(f_neg), treating 0×log2(0) = 0
+    - Compute volume_factor = 1 - exp(-E_total / 3.0)
+    - Return H_conflict × volume_factor, bounded in [0.0, 1.0]
+    - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7_
+
+  - [x] 5.2 Write property tests for contradiction (`tests/test_pbt_v3_posterior.py`)
+    - **Property 8: Contradiction is zero when evidence is unidirectional**
+    - **Property 9: Contradiction score is bounded in [0, 1]**
+    - **Validates: Requirements 7.6, 7.7, 21.7**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 5.3 Write unit tests for LLR entropy contradiction (`tests/test_v3_contradiction.py`)
+    - Test all bullish clusters → contradiction = 0.0
+    - Test equal split of evidence → high contradiction near 1.0
+    - Test E_total = 0 → contradiction = 0.0
+    - Test volume_factor growth: small evidence mass → suppressed score
+    - _Requirements: 7.1–7.7_
+
+- [x] 6. Replace confidence formula
+
+  - [x] 6.1 Implement multiplicative confidence v3 in `services/aggregation/worker.py`
+    - Add `compute_v3_confidence(n_eff_total, q_values, llrs, strength, regime_confidence_mult, contradiction, data_quality) -> float`
+    - Compute C_evidence = 1 - exp(-n_eff_total / 5.0)
+    - Compute C_quality = weighted_mean(q_i, weight=|LLR_i|)
+    - Compute confidence = clamp(C_evidence × sqrt(C_quality) × sqrt(max(strength, 0.05)) × regime_confidence_mult × (1 - contradiction) × data_quality, 0, 1)
+    - _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5_
+
+  - [x] 6.2 Implement data quality v3 in `services/aggregation/worker.py`
+    - Add `compute_v3_data_quality(units, extraction_failure_rate, age_newest_hours, n_source_types) -> float`
+    - Q_parse = 1 - extraction_failure_rate
+    - Q_conf = weighted_mean(extraction_conf_i, weight=impact_i)
+    - Q_fresh = exp(-age_newest_hours / 168)
+    - Q_coverage = 1 - exp(-N_valid / 5)
+    - Q_diversity = min(1, log2(1 + N_source_types) / log2(4))
+    - data_quality_score = clamp(Q_parse × sqrt(Q_conf) × Q_fresh × Q_coverage × Q_diversity, 0, 1)
+    - Force informational mode when data_quality < 0.50 OR N_valid < 2 OR Q_parse < 0.50
+    - Force informational when only macro/competitive evidence unless macro_only_enabled
+    - _Requirements: 17.1, 17.2, 17.3, 17.4, 17.5, 17.6, 17.7, 17.8_
+
+  - [x] 6.3 Write property tests for confidence and data quality (`tests/test_pbt_v3_posterior.py`)
+    - **Property 10: Multiplicative confidence is bounded in [0, 1] and suppressed by weak dimensions**
+    - **Property 17: Data quality score is bounded in [0, 1]**
+    - **Validates: Requirements 8.3, 8.5, 17.6, 21.6**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 6.4 Write unit tests for confidence and data quality (`tests/test_v3_confidence.py`)
+    - Test zero data quality → near-zero confidence
+    - Test full contradiction (1.0) → zero confidence
+    - Test low n_eff → suppressed C_evidence
+    - Test data quality boundary cases (Q_parse < 0.50 → informational)
+    - _Requirements: 8.1–8.5, 17.1–17.8_
+
+- [x] 7. Checkpoint - Verify core pipeline
+  - Ensure all tests pass for posterior, contradiction, confidence, and data quality.
+  - Ensure all tests pass, ask the user if questions arise.
+
+- [x] 8. Convert macro and competitive layers to emit LLR
+
+  - [x] 8.1 Implement noisy-OR macro exposure and LLR emission in `services/aggregation/interpolation.py`
+    - Add `compute_normalized_macro_exposure(overlaps: dict[str, float]) -> float`
+    - E_raw = 1 - product(1 - w_k × O_k) with weights: w_geo=0.35, w_supply=0.25, w_commodity=0.25, w_sector=0.15
+    - E_max = 1 - product(1 - w_k)
+    - E_macro = E_raw / E_max (normalized to [0, 1])
+    - Apply resilience dampener per tier: global_leader=0.70, multinational=0.85, regional=1.00, domestic=1.20
+    - Add `compute_macro_llr(macro_impact, event_confidence, q_recency, macro_direction) -> float`
+    - p_macro = clamp(0.50 + 0.30 × macro_impact × event_confidence × q_recency, 0.501, 0.80)
+    - LLR_macro = macro_direction × ln(p_macro / (1 - p_macro))
+    - Feed macro LLR into shared posterior without separate post-hoc modifier
+    - _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5_
+
+  - [x] 8.2 Implement correlation-shrunk competitive propagation in `services/aggregation/signal_propagation.py`
+    - Add `compute_shrunk_correlation(rho_rolling, n_observations, same_sector) -> float`
+    - rho_prior = 0.30 (same_sector) or 0.10 (cross_sector)
+    - rho_shrunk = (n/(n+30)) × rho_rolling + (30/(n+30)) × rho_prior
+    - rho_effective = max(rho_shrunk, 0)
+    - Add `compute_competitive_llr(llr_source, rho_effective, d_network, pattern_confidence) -> float`
+    - attenuation = rho_effective × exp(-0.85 × d_network), max_distance = 3
+    - LLR_competitive = clamp(llr_source × attenuation × pattern_confidence, -1.25, 1.25)
+    - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5_
+
+  - [x] 8.3 Write property tests for macro and competitive layers (`tests/test_pbt_v3_layers.py`)
+    - **Property 13: Noisy-OR normalized exposure is bounded in [0, 1]**
+    - **Property 14: Competitive LLR is clamped to [-1.25, 1.25]**
+    - **Property 15: Graph attenuation is zero beyond max distance**
+    - **Validates: Requirements 9.1, 9.2, 10.3, 10.4, 10.5**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 8.4 Write unit tests for macro and competitive layers (`tests/test_v3_layers.py`)
+    - Test noisy-OR: all O_k = 1.0 → E_macro = 1.0; all O_k = 0 → E_macro = 0
+    - Test resilience dampener per tier
+    - Test macro LLR at boundary values
+    - Test shrunk correlation convergence (n → ∞ approaches rho_rolling)
+    - Test competitive LLR clamp at ±1.25
+    - Test distance > 3 → zero attenuation
+    - _Requirements: 9.1–9.5, 10.1–10.5_
+
+- [x] 9. Replace EV gate with expected-return distribution
+
+  - [x] 9.1 Implement posterior state projection in `services/aggregation/projection.py`
+    - Add `V3ProjectionState` dataclass (a_t, p_up_projected, projected_strength, diverges, phi_regime)
+    - Implement `compute_v3_projection(a_prev, cluster_llrs, regime, p_prior, projection_horizon, known_catalyst_llr=0.0)`
+    - Evidence state: A_t = phi_regime × A_{t-1} + sum(LLR_c), init A_0 = 0.0
+    - Regime decay: panic=0.35, trend_following=0.80, mean_reversion=0.55, uncertainty=0.50
+    - Projected alpha: A_projected = phi^h × A_t + known_catalyst_LLR
+    - P_up_projected = sigmoid(logit(P_prior) + A_projected)
+    - Projected strength = abs(2 × P_up_projected - 1)
+    - Flag divergence when sign(P_up_projected - 0.5) ≠ sign(P_up_t - 0.5)
+    - _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7_
+
+  - [x] 9.2 Implement return distribution and EV gate in `services/recommendation/eligibility.py`
+    - Add `ReturnDistribution` dataclass (sigma_h, mu_h, ev_long, min_edge, eligible)
+    - Add `compute_return_distribution(a_projected, confidence, realized_vol_20d, horizon_days, costs, regime)`
+    - sigma_h = realized_vol_20d × sqrt(horizon_days / 252); default vol = 0.25 if unavailable
+    - mu_h = tanh(A_projected / 3.0) × confidence × sigma_h
+    - CVaR_5 = sigma_h × 1.645 × 1.4
+    - EV_long = mu_h - costs - 0.10 × CVaR_5
+    - Regime min_edge: panic=0.0100, trend_following=0.0035, mean_reversion=0.0050, uncertainty=0.0075
+    - Eligibility: EV_long > min_edge AND EV_long > max(0.0025, 0.25 × costs)
+    - Also require: confidence >= regime_confidence_min, contradiction <= regime_contradiction_max, n_eff_total >= 2.0, data_quality >= 0.50
+    - _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 12.7_
+
+  - [x] 9.3 Implement regime-aware eligibility and mode escalation in `services/recommendation/eligibility.py`
+    - Add regime-specific eligibility thresholds: panic (conf≥0.70, contra≤0.25, str≥0.36), trend_following (conf≥0.55, contra≤0.40, str≥0.20), mean_reversion (conf≥0.60, contra≤0.35, str≥0.26), uncertainty (conf≥0.65, contra≤0.30, str≥0.30)
+    - Action mapping: BUY when P_up >= bullish_threshold and EV > min_edge; SELL when existing position and EV_exit > EV_hold; HOLD when existing; WATCH otherwise
+    - live_eligible: BUY/SELL + conf >= 0.75 + contra <= 0.20 + n_eff >= 5 + EV > 2×min_edge + risk_passed
+    - paper_eligible: BUY/SELL + conf >= 0.60 + EV > min_edge + risk_passed
+    - Otherwise: informational
+    - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5_
+
+  - [x] 9.4 Write property tests for projection (`tests/test_pbt_v3_projection.py`)
+    - **Property 16: Projection evidence state decays toward zero**
+    - **Validates: Requirements 11.1, 11.3**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 9.5 Write unit tests for EV gate and eligibility (`tests/test_v3_eligibility.py`)
+    - Test EV_long positive → eligible
+    - Test EV_long negative → ineligible
+    - Test regime-specific min_edge thresholds
+    - Test mode escalation: live vs paper vs informational
+    - Test projection decay convergence
+    - Test divergence flag behavior
+    - _Requirements: 11.1–11.7, 12.1–12.7, 13.1–13.5_
+
+- [x] 10. Checkpoint - Verify decision layer
+  - Ensure all tests pass for projection, EV gate, eligibility, and layer integrations.
+  - Ensure all tests pass, ask the user if questions arise.
+
+- [x] 11. Replace sizing with fractional Kelly under existing risk caps
+
+  - [x] 11.1 Implement fractional Kelly position sizing in `services/trading/position_sizer.py`
+    - Add `compute_kelly_sizing(p_win, b, confidence, data_quality, contradiction, max_position_pct, available_caps) -> float`
+    - Compute reward ratio b = clamp(1.2 + 2.0 × confidence + 1.0 × strength - contradiction, 1.2, 3.0)
+    - Compute f_kelly = (p_win × b - (1 - p_win)) / b
+    - portfolio_pct = clamp(max(0, f_kelly) × 0.25 × confidence × data_quality × (1 - contradiction), 0, max_position_pct)
+    - Apply min of: max_position_pct, sector_capacity, correlation_capacity (0 if avg corr > 0.80), heat_capacity
+    - If portfolio_pct < 0.005 → downgrade to WATCH (reason: position_below_minimum)
+    - If f_kelly <= 0 → portfolio_pct = 0, downgrade to WATCH (reason: negative_edge)
+    - _Requirements: 14.1, 14.2, 14.3, 14.4, 14.5, 14.6, 14.7_
+
+  - [x] 11.2 Implement regime-aware stop loss and take profit in `services/trading/stop_loss_manager.py`
+    - Add v3 stop computation: stop_distance_pct = max(ATR_pct × regime_ATR_mult, sigma_h × 1.25, 0.005)
+    - stop_loss = entry_price × (1 - stop_distance_pct)
+    - take_profit = entry_price × (1 + b × stop_distance_pct) where b = reward ratio
+    - Activate trailing stop when unrealized_gain >= 0.50 × TP distance
+    - trailing_stop = max(existing_stop, current_price × (1 - trailing_distance_pct))
+    - trailing_distance_pct = max(ATR_pct × trailing_ATR_mult, sigma_h × 0.75)
+    - Trailing stop must be monotonically non-decreasing
+    - _Requirements: 16.1, 16.2, 16.3, 16.4, 16.5_
+
+  - [x] 11.3 Write property tests for Kelly sizing and stops (`tests/test_pbt_v3_decision.py`)
+    - **Property 11: Fractional Kelly sizing is bounded and respects negative edge**
+    - **Property 18: Stop loss is below entry price and take profit is above**
+    - **Property 19: Trailing stop never decreases**
+    - **Validates: Requirements 14.4, 14.7, 16.2, 16.3, 16.5, 21.8, 21.9**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 11.4 Write unit tests for Kelly sizing and stops (`tests/test_v3_sizing.py`)
+    - Test p_win=0.3, b=2 → f_kelly < 0 → size = 0
+    - Test p_win=0.7, b=2 → positive size within caps
+    - Test cap enforcement (sector, correlation, heat)
+    - Test position_below_minimum downgrade
+    - Test stop/TP computation with known inputs
+    - Test trailing stop monotonicity over a price sequence
+    - _Requirements: 14.1–14.7, 16.1–16.5_
+
+- [x] 12. Replace portfolio heat with stop-defined risk dollars
+
+  - [x] 12.1 Implement stop-defined portfolio heat in `services/risk/engine.py`
+    - Add `compute_portfolio_heat(positions, stop_distances) -> float`
+    - risk_dollars = position_value × stop_distance_pct for each position
+    - portfolio_heat = sum(risk_dollars)
+    - Add `check_heat_capacity(current_heat, new_risk_dollars, max_heat_pct, portfolio_value) -> bool`
+    - Reject new entry when current_heat + new_risk_dollars > max_heat_pct × portfolio_value
+    - Integrate available_heat_capacity into Kelly sizing pipeline
+    - _Requirements: 15.1, 15.2, 15.3, 15.4, 15.5_
+
+  - [x] 12.2 Implement risk tier auto-adjustment v3 in `services/risk/engine.py`
+    - Add `TierMetrics` dataclass (profit_factor_30d, max_drawdown_30d, calibration_error, realized_sharpe_30d, n_trades_30d, reserve_pool_pct)
+    - Add `evaluate_tier_adjustment(metrics) -> str` returning 'upgrade'|'downgrade'|'hold'
+    - Downgrade if ANY: profit_factor < 1.0 OR drawdown > 0.12 OR calibration_error > 0.20 OR sharpe < 0
+    - Upgrade only if ALL: profit_factor > 1.35 AND drawdown < 0.05 AND calibration_error < 0.12 AND reserve > 0.20 AND N_trades >= 20
+    - Apply downgrade immediately; enforce 7-day upgrade cooldown
+    - Evaluate once per calendar day after session close
+    - _Requirements: 18.1, 18.2, 18.3, 18.4, 18.5, 18.6_
+
+  - [x] 12.3 Write property tests for heat and tier adjustment (`tests/test_pbt_v3_decision.py`)
+    - **Property 22: Portfolio heat rejection is correct**
+    - **Property 23: Tier auto-adjustment obeys downgrade-any, upgrade-all logic**
+    - **Validates: Requirements 15.3, 15.5, 18.3, 18.4**
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 12.4 Write unit tests for heat and tier (`tests/test_v3_risk.py`)
+    - Test heat computation: 3 positions with known stops → expected heat
+    - Test heat rejection: heat at limit → new entry blocked
+    - Test tier downgrade: single bad metric triggers downgrade
+    - Test tier upgrade: all metrics good → upgrade
+    - Test tier upgrade: one metric bad → hold (not upgrade)
+    - Test 7-day cooldown enforcement
+    - _Requirements: 15.1–15.5, 18.1–18.6_
+
+- [x] 13. Checkpoint - Verify sizing and risk layer
+  - Ensure all tests pass for Kelly sizing, stops, heat, and tier adjustment.
+  - Ensure all tests pass, ask the user if questions arise.
+
+- [x] 14. Retire heuristic scoring to explainability-only mode
+
+  - [x] 14.1 Wire v3 pipeline end-to-end in `services/aggregation/worker.py`
+    - Orchestrate full pipeline: EvidenceUnit → q_i → LLR → cluster → posterior → contradiction → confidence → data_quality
+    - Store v3 posterior in TrendSummary JSONB metadata (p_up, log_odds, strength, confidence, contradiction, n_eff, data_quality, regime)
+    - Store return model in Recommendation JSONB metadata (mu_h, sigma_h, ev_long, min_edge)
+    - Store explainability payload (top_positive_clusters, top_negative_clusters, suppression_reasons, risk_adjustments)
+    - Preserve WeightedSignal as intermediate representation before LLR conversion
+    - _Requirements: 20.1, 20.2, 20.3, 20.4, 20.5_
+
+  - [x] 14.2 Retain heuristic pipeline as fallback with explainability overlay
+    - Keep existing heuristic scoring path fully functional (no removal)
+    - Mark heuristic outputs with `pipeline_mode: "heuristic"` in metadata
+    - Ensure heuristic mode still produces valid TrendSummary and Recommendation objects
+    - Test feature flag toggle: v3 → heuristic → v3 round-trip
+    - _Requirements: 19.1, 19.2, 19.5, 20.4_
+
+  - [x] 14.3 Write property test for JSON round-trip (`tests/test_pbt_v3_posterior.py`)
+    - **Property 12: Posterior state JSON round-trip**
+    - **Validates: Requirements 20.1, 21.10**
+    - Serialize V3Posterior to JSON and deserialize, verify equivalence within 1e-10
+    - Use Hypothesis with `@settings(max_examples=100)`
+
+  - [x] 14.4 Write integration tests for full pipeline (`tests/test_v3_integration.py`)
+    - Test full path: raw signals → EvidenceUnit → q_i → LLR → cluster → posterior → recommendation
+    - Test feature flag false → heuristic path, flag true → v3 path
+    - Test v3 exception → heuristic fallback + error logged
+    - Test output JSONB contains expected v3 fields
+    - _Requirements: 19.1–19.6, 20.1–20.5_
+
+- [x] 15. Final checkpoint - All tests green
+  - Ensure all tests pass, ask the user if questions arise.
+  - Run full test suite: `.venv/bin/python -m pytest tests/test_pbt_v3_*.py tests/test_v3_*.py -x --tb=short -q`
+  - Verify no regressions in existing heuristic pipeline tests
+
+## Notes
+
+- Tasks marked with `*` are optional and can be skipped for faster MVP
+- Each task references specific requirements for traceability
+- Checkpoints ensure incremental validation after each logical phase
+- Property tests validate the 23 correctness properties defined in the design
+- Unit tests validate specific examples, edge cases, and error handling
+- The implementation preserves the existing heuristic pipeline as a fully functional fallback
+- All v3 code is gated behind `v3_engine_enabled` — no changes to production behavior until flag is flipped
+- Run tests with: `.venv/bin/python -m pytest tests/ -x --tb=short -q`
+- Property tests use Hypothesis: `@settings(max_examples=100)`
+
+## Task Dependency Graph
+
+```json
+{
+  "waves": [
+    { "id": 0, "tasks": ["1.1"] },
+    { "id": 1, "tasks": ["1.2", "1.4"] },
+    { "id": 2, "tasks": ["1.3"] },
+    { "id": 3, "tasks": ["1.5", "1.6"] },
+    { "id": 4, "tasks": ["2.1"] },
+    { "id": 5, "tasks": ["2.2"] },
+    { "id": 6, "tasks": ["2.3"] },
+    { "id": 7, "tasks": ["2.4", "2.5"] },
+    { "id": 8, "tasks": ["4.1"] },
+    { "id": 9, "tasks": ["4.2"] },
+    { "id": 10, "tasks": ["4.3", "4.4"] },
+    { "id": 11, "tasks": ["5.1"] },
+    { "id": 12, "tasks": ["5.2", "5.3"] },
+    { "id": 13, "tasks": ["6.1", "6.2"] },
+    { "id": 14, "tasks": ["6.3", "6.4"] },
+    { "id": 15, "tasks": ["8.1", "8.2"] },
+    { "id": 16, "tasks": ["8.3", "8.4"] },
+    { "id": 17, "tasks": ["9.1"] },
+    { "id": 18, "tasks": ["9.2", "9.3"] },
+    { "id": 19, "tasks": ["9.4", "9.5"] },
+    { "id": 20, "tasks": ["11.1", "11.2"] },
+    { "id": 21, "tasks": ["11.3", "11.4"] },
+    { "id": 22, "tasks": ["12.1", "12.2"] },
+    { "id": 23, "tasks": ["12.3", "12.4"] },
+    { "id": 24, "tasks": ["14.1"] },
+    { "id": 25, "tasks": ["14.2"] },
+    { "id": 26, "tasks": ["14.3", "14.4"] }
+  ]
+}
+```

.kiro/specs/ops-pipeline-fixes/.config.kiro

@@ -0,0 +1 @@
+{"specId": "f5d99301-94ef-4dc2-8ba4-ccefeee7ecba", "workflowType": "requirements-first", "specType": "bugfix"}

.kiro/specs/ops-pipeline-fixes/bugfix.md

@@ -0,0 +1,73 @@
+# Bugfix Requirements Document
+
+## Introduction
+
+Multiple operational bugs discovered in the stonks-beta namespace prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput. The core issue is that the outcome evaluation → metrics computation → quality gate pipeline is completely disconnected from the production scheduler, making the platform unable to self-calibrate or validate predictions. Additionally, Polygon API rate limiting causes ~40% request failures per cycle, a broken config query prevents the v3 engine from being toggled, several periodic snapshot tasks are missing from the scheduler, the lake-publisher deployment is idle/redundant, and order rejection reasons are lost.
+
+## Bug Analysis
+
+### Current Behavior (Defect)
+
+1.1 WHEN the scheduler enqueues ingestion jobs for all 50 tickers' news_api and market_api sources simultaneously THEN the system exhausts the Polygon free-tier rate limit (5 req/min) resulting in ~40% of sources receiving HTTP 429 Too Many Requests every cycle
+
+1.2 WHEN the aggregation worker reads the v3_engine_enabled flag via `_V3_ENGINE_FLAG_QUERY` THEN the system queries non-existent columns `key` and `value` on the `risk_configs` table (actual schema: `name` varchar, `config` JSONB) causing a PostgreSQL error every aggregation cycle
+
+1.3 WHEN a scheduler cycle completes THEN the system never calls `evaluate_matured_predictions()` because it is not wired into the scheduler's main loop — only imported in `backtest_replay.py`
+
+1.4 WHEN a scheduler cycle completes THEN the system never calls `compute_and_store_metric_snapshots()` because it is not wired into the scheduler's main loop — only called from backtest replay
+
+1.5 WHEN the model quality gate evaluates trading eligibility THEN the system always fails with "no model metric snapshot available — defaulting to paper-only" because `model_metric_snapshots` table is permanently empty (consequence of bug 1.4)
+
+1.6 WHEN the trading engine runs daily THEN the system never captures portfolio state snapshots to the `portfolio_snapshots` table because no periodic scheduler task invokes this capture
+
+1.7 WHEN the trading engine runs daily THEN the system never captures risk state snapshots to the `daily_risk_snapshots` table because no periodic scheduler task invokes this capture
+
+1.8 WHEN a prediction snapshot is created while Polygon rate-limiting has prevented the market data fetch THEN the system stores NULL in `price_at_prediction` (affecting 21% of snapshots), degrading downstream outcome evaluation accuracy
+
+1.9 WHEN the standalone `lake-publisher` deployment polls `stonks:beta:queue:lake_publish` THEN the queue is always empty (0 items) because all lake publishing happens inline in broker-adapter and recommendation services — the deployment consumes zero work and wastes resources
+
+1.10 WHEN Alpaca returns HTTP 401 for an order submission THEN the system sets order status to "rejected" but leaves the `rejection_reason` column NULL, capturing the error message only in the `decision_trace` JSONB field
+
+### Expected Behavior (Correct)
+
+2.1 WHEN the scheduler enqueues ingestion jobs for Polygon-backed sources (news_api, market_api) THEN the system SHALL pace/stagger requests across the polling interval to stay within the Polygon rate limit, achieving near-zero 429 responses per cycle
+
+2.2 WHEN the aggregation worker reads the v3_engine_enabled flag THEN the system SHALL query `SELECT config FROM risk_configs WHERE name = 'v3_engine_enabled'` and parse the JSONB value to determine the boolean toggle state
+
+2.3 WHEN a scheduler cycle completes and sufficient time has elapsed since the last evaluation THEN the system SHALL call `evaluate_matured_predictions()` to evaluate prediction snapshots whose horizon has elapsed, populating the `prediction_outcomes` table
+
+2.4 WHEN a scheduler cycle completes and sufficient time has elapsed since the last computation THEN the system SHALL call `compute_and_store_metric_snapshots()` to compute aggregate model metrics across all lookback/horizon combinations, populating `model_metric_snapshots`
+
+2.5 WHEN the model quality gate evaluates trading eligibility THEN the system SHALL have recent metric snapshots available and evaluate thresholds against actual model performance data
+
+2.6 WHEN market hours close (or on a daily schedule) THEN the system SHALL capture and persist the current portfolio state to `portfolio_snapshots` including value, returns, positions, and risk metrics
+
+2.7 WHEN market hours close (or on a daily schedule) THEN the system SHALL capture and persist the current risk state to `daily_risk_snapshots` including portfolio value, daily P&L, trade count, and sector positions
+
+2.8 WHEN a prediction snapshot is created and market price is unavailable due to rate limiting THEN the system SHALL retry the price fetch or defer the snapshot until price data is available, reducing NULL `price_at_prediction` occurrences to near zero
+
+2.9 WHEN the lake-publisher deployment architecture is reviewed THEN the system SHALL either route lake publish jobs through the Redis queue to the standalone deployment, or remove the redundant deployment — eliminating the idle pod
+
+2.10 WHEN Alpaca returns an HTTP error (401, 403, or any rejection) for an order submission THEN the system SHALL populate the `rejection_reason` column with the HTTP error message/status in addition to recording it in `decision_trace`
+
+### Unchanged Behavior (Regression Prevention)
+
+3.1 WHEN sources with valid rate-limit headroom are enqueued THEN the system SHALL CONTINUE TO enqueue and process them without artificial delay
+
+3.2 WHEN risk_configs is queried for other configuration keys (e.g., `model_quality_gate_config`, `macro_enabled`) THEN the system SHALL CONTINUE TO read them correctly using the existing `name`/`config` column pattern
+
+3.3 WHEN the backtest replay module calls `evaluate_matured_predictions()` and `compute_and_store_metric_snapshots()` THEN the system SHALL CONTINUE TO execute them as part of backtest validation
+
+3.4 WHEN prediction snapshots are created with available market prices THEN the system SHALL CONTINUE TO store the correct `price_at_prediction` value immediately
+
+3.5 WHEN the existing inline lake publishing in broker-adapter and recommendation services writes facts THEN the system SHALL CONTINUE TO produce correct Parquet partitions in MinIO
+
+3.6 WHEN orders succeed (HTTP 200 from Alpaca) THEN the system SHALL CONTINUE TO process them normally without modifying the `rejection_reason` column
+
+3.7 WHEN the scheduler runs ingestion, extraction, aggregation, recommendation, and trading tasks THEN the system SHALL CONTINUE TO execute them on the existing cadence without disruption
+
+3.8 WHEN the trading engine makes decisions and submits orders THEN the system SHALL CONTINUE TO record full decision context in `decision_trace` JSONB as before
+
+3.9 WHEN the model quality gate passes (once metrics are populated) THEN the system SHALL CONTINUE TO allow promotion to live trading mode per existing threshold logic
+
+3.10 WHEN the reporting collector fetches portfolio_snapshots and daily_risk_snapshots for report generation THEN the system SHALL CONTINUE TO query and render them using the existing schema

.kiro/specs/ops-pipeline-fixes/design.md

@@ -0,0 +1,396 @@
+# Technical Design: ops-pipeline-fixes
+
+## Overview
+
+This design addresses 10 operational bugs that prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput in the `stonks-beta` namespace. The fixes span the scheduler (rate limiting + periodic tasks), aggregation worker (config query), broker service (rejection reason), prediction snapshot (price fallback), and Helm chart (dead pod removal). All changes are localized with graceful fallbacks and no schema migrations required.
+
+## Bug Details
+
+Multiple operational bugs in the `stonks-beta` namespace prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput. The core pipeline (ingestion → extraction → aggregation → recommendation → trading) flows end-to-end, but:
+- The outcome evaluation → metrics computation → quality gate feedback loop is completely disconnected
+- Polygon API rate limiting causes ~40% ingestion failures per cycle
+- A broken config query prevents the v3 engine toggle from working
+- Portfolio/risk snapshots are never captured
+- Order rejection reasons are lost
+
+```mermaid
+graph TD
+    subgraph "Scheduler (services/scheduler/app.py)"
+        A[schedule_cycle] -->|paced enqueue| B[Ingestion Queue]
+        C[validation_cycle] -->|hourly| D[evaluate_matured_predictions]
+        C -->|after outcomes| E[compute_and_store_metric_snapshots]
+        F[snapshot_cycle] -->|daily 16:30 ET| G[capture_portfolio_snapshot]
+        F -->|daily 16:30 ET| H[capture_risk_snapshot]
+    end
+
+    subgraph "Aggregation (services/aggregation/worker.py)"
+        I[_read_v3_flag] -->|fixed query| J[risk_configs.config JSONB]
+    end
+
+    subgraph "Broker (services/adapters/broker_service.py)"
+        K[persist_order] -->|rejected status| L[orders.rejection_reason]
+    end
+
+    D --> O[prediction_outcomes]
+    E --> P[model_metric_snapshots]
+    P --> Q[Quality Gate]
+```
+
+## Expected Behavior
+
+2.1 The scheduler SHALL pace Polygon API requests within the free-tier limit (~5 req/min), achieving near-zero 429 responses per cycle.
+
+2.2 The aggregation worker SHALL read `v3_engine_enabled` from the `risk_configs` JSONB `config` column (not non-existent `key`/`value` columns).
+
+2.3 The scheduler SHALL call `evaluate_matured_predictions()` hourly to populate `prediction_outcomes`.
+
+2.4 The scheduler SHALL call `compute_and_store_metric_snapshots()` after outcome evaluation to populate `model_metric_snapshots`.
+
+2.5 The quality gate SHALL have recent metric data available once the validation cycle runs.
+
+2.6 The scheduler SHALL capture daily portfolio snapshots to `portfolio_snapshots` after market close.
+
+2.7 The scheduler SHALL capture daily risk snapshots to `daily_risk_snapshots` after market close.
+
+2.8 Prediction snapshots SHALL fall back to positions table prices when market_snapshots data is unavailable.
+
+2.9 The lake-publisher deployment SHALL be scaled to 0 (idle pod, wasted resources).
+
+2.10 The broker service SHALL populate `rejection_reason` on orders when broker or risk engine rejects.
+
+## Hypothesized Root Cause
+
+### Bug 1.1 — Polygon Rate Limiting
+`POLYGON_GLOBAL_RATE_LIMIT = 45` in `services/scheduler/app.py` is set for a paid Polygon plan but the deployed instance uses the free tier (5 req/min). All 50+ sources are attempted per cycle, exhausting the limit instantly.
+
+### Bug 1.2 — v3_engine_enabled Config Read
+`_V3_ENGINE_FLAG_QUERY` in `services/aggregation/worker.py` reads `SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'`. The actual table has columns `name` (varchar) and `config` (JSONB) — no `key` or `value` column exists.
+
+### Bug 1.3 & 1.4 — Outcome Evaluator & Metrics Never Scheduled
+`evaluate_matured_predictions()` and `compute_and_store_metric_snapshots()` exist in `services/validation/` but are only imported in `services/trading/backtest_replay.py`. The scheduler main loop in `services/scheduler/app.py` has no call to either function.
+
+### Bug 1.5 — Quality Gate Permanently Failing
+`services/trading/model_quality_gate.py` queries `model_metric_snapshots` which is always empty (consequence of 1.4). Returns "no model metric snapshot available — defaulting to paper-only" every time.
+
+### Bug 1.6 & 1.7 — Portfolio/Risk Snapshots
+The trading engine has `_persist_daily_snapshot()` but it only executes when the engine's main loop is actively processing trades. The trading-engine pod shows only health checks — its main loop isn't cycling because there are no active trade triggers flowing through it. No fallback capture exists in the scheduler.
+
+### Bug 1.8 — Market Price Gaps
+`services/validation/prediction_snapshot.py` queries `market_snapshots` for price at prediction time. When Polygon rate limiting prevents market data fetches, no snapshot exists and `price_at_prediction` is NULL. 21% of snapshots affected.
+
+### Bug 1.9 — Lake Publisher Idle
+The standalone `lake-publisher` deployment polls `stonks:beta:queue:lake_publish` but all services (broker-adapter, recommendation) import `services.lake_publisher.worker` directly and publish inline — never pushing to the Redis queue.
+
+### Bug 1.10 — Order rejection_reason NULL
+`_INSERT_ORDER` SQL in `services/adapters/broker_service.py` doesn't include `rejection_reason` or `rejected_at` columns. The error is stored in `decision_trace` JSONB but the dedicated column stays NULL. The reconciliation path (`_reconcile_open_orders`) does set these columns, but initial persist does not.
+
+## Fix Implementation
+
+### Fix 1: Polygon Rate Limit Constant (Bug 1.1)
+
+**File:** `services/scheduler/app.py`
+
+Replace the hardcoded constant with an env-configurable value defaulting to 5:
+
+```python
+# Before:
+POLYGON_GLOBAL_RATE_LIMIT: int = 45
+
+# After:
+POLYGON_GLOBAL_RATE_LIMIT: int = int(os.getenv("POLYGON_GLOBAL_RATE_LIMIT", "5"))
+```
+
+The existing `check_rate_limit()` function already implements per-minute windowed counting and skips sources once the limit is hit. By reducing the constant to match the free-tier limit, the system will naturally pace — enqueuing ~5 Polygon sources per minute across scheduler ticks (15s interval = 4 ticks/min). Skipped sources are retried next cycle.
+
+**Validates:** Bugfix 2.1; Regression 3.1, 3.7
+
+---
+
+### Fix 2: v3_engine_enabled Config Query (Bug 1.2)
+
+**File:** `services/aggregation/worker.py`
+
+Replace the broken query and function:
+
+```python
+# Before:
+_V3_ENGINE_FLAG_QUERY = """
+SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'
+"""
+
+# After:
+_V3_ENGINE_FLAG_QUERY = """
+SELECT config->>'v3_engine_enabled' AS enabled
+FROM risk_configs
+WHERE name = 'default' AND active = TRUE
+LIMIT 1
+"""
+
+async def _read_v3_flag(pool: asyncpg.Pool) -> bool:
+    """Read v3_engine_enabled from risk_configs JSONB. Default False on error."""
+    try:
+        row = await pool.fetchrow(_V3_ENGINE_FLAG_QUERY)
+        if row and row["enabled"]:
+            return row["enabled"].lower() in ("true", "1", "yes")
+        return False
+    except Exception as e:
+        logger.warning("Failed to read v3_engine_enabled flag: %s", e)
+        return False
+```
+
+Reads from the `default` active risk_config's JSONB `config` field. Falls back to False (unchanged fail-safe).
+
+**Validates:** Bugfix 2.2; Regression 3.2
+
+---
+
+### Fix 3: Validation Cycle in Scheduler (Bugs 1.3, 1.4, 1.5)
+
+**File:** `services/scheduler/app.py`
+
+Add a new periodic task (every ~240 ticks = ~60 minutes):
+
+```python
+# New constant:
+VALIDATION_CYCLE_INTERVAL = int(os.getenv("VALIDATION_CYCLE_INTERVAL", "240"))
+
+# New counter in main():
+validation_counter = 0
+
+# In main loop after existing periodic tasks:
+validation_counter += 1
+if validation_counter >= VALIDATION_CYCLE_INTERVAL:
+    validation_counter = 0
+    await run_validation_cycle(pool)
+```
+
+New function:
+
+```python
+async def run_validation_cycle(pool: asyncpg.Pool) -> None:
+    """Run outcome evaluation and metric computation (hourly).
+
+    Requirements: 2.3, 2.4, 2.5
+    """
+    from services.validation.outcome_evaluator import evaluate_matured_predictions
+    from services.validation.metrics import compute_and_store_metric_snapshots
+
+    try:
+        outcomes = await evaluate_matured_predictions(pool)
+        logger.info("Validation: evaluated %d prediction outcomes", outcomes)
+    except Exception:
+        logger.exception("Validation: outcome evaluation failed")
+        return  # Skip metrics if outcomes failed
+
+    try:
+        snapshots = await compute_and_store_metric_snapshots(pool)
+        logger.info("Validation: computed %d metric snapshots", len(snapshots))
+    except Exception:
+        logger.exception("Validation: metric computation failed")
+```
+
+**Validates:** Bugfix 2.3, 2.4, 2.5; Regression 3.3
+
+---
+
+### Fix 4: Daily Portfolio & Risk Snapshots (Bugs 1.6, 1.7)
+
+**File:** `services/scheduler/app.py`
+
+Add a daily snapshot task that runs every ~60 minutes but only captures once per day after 16:30 ET:
+
+```python
+SNAPSHOT_CYCLE_INTERVAL = int(os.getenv("SNAPSHOT_CYCLE_INTERVAL", "240"))
+snapshot_counter = 0
+
+# In main loop:
+snapshot_counter += 1
+if snapshot_counter >= SNAPSHOT_CYCLE_INTERVAL:
+    snapshot_counter = 0
+    await maybe_capture_daily_snapshots(pool)
+```
+
+New function:
+
+```python
+async def maybe_capture_daily_snapshots(pool: asyncpg.Pool) -> None:
+    """Capture portfolio and risk snapshots once daily after market close.
+
+    Requirements: 2.6, 2.7
+    """
+    et_now = datetime.now(ZoneInfo("America/New_York"))
+
+    # Only after 4:30 PM ET
+    if et_now.hour < 16 or (et_now.hour == 16 and et_now.minute < 30):
+        return
+
+    today = et_now.date()
+
+    # Already captured today?
+    existing = await pool.fetchval(
+        "SELECT 1 FROM portfolio_snapshots WHERE snapshot_date = $1 LIMIT 1",
+        today,
+    )
+    if existing:
+        return
+
+    # Portfolio snapshot from positions + account data
+    try:
+        positions = await pool.fetch("SELECT * FROM positions WHERE quantity > 0")
+        portfolio_value = sum(
+            float(r["current_price"] or 0) * float(r["quantity"])
+            for r in positions
+        )
+        unrealized_pnl = sum(float(r["unrealized_pnl"] or 0) for r in positions)
+
+        await pool.execute(
+            """INSERT INTO portfolio_snapshots
+               (snapshot_date, portfolio_value, unrealized_pnl, positions)
+               VALUES ($1, $2, $3, $4::jsonb)""",
+            today, portfolio_value, unrealized_pnl,
+            json.dumps([dict(r) for r in positions], default=str),
+        )
+        logger.info("Captured portfolio snapshot: value=%.2f", portfolio_value)
+    except Exception:
+        logger.exception("Failed to capture portfolio snapshot")
+
+    # Risk snapshot from daily activity
+    try:
+        daily_orders = await pool.fetchval(
+            "SELECT count(*) FROM orders WHERE created_at::date = $1", today
+        )
+        daily_pnl = sum(float(r["unrealized_pnl"] or 0) for r in positions) if positions else 0.0
+
+        await pool.execute(
+            """INSERT INTO daily_risk_snapshots
+               (account_id, snapshot_date, portfolio_value, daily_pnl, daily_trade_count)
+               VALUES ((SELECT id FROM broker_accounts LIMIT 1), $1, $2, $3, $4)
+               ON CONFLICT DO NOTHING""",
+            today, portfolio_value, daily_pnl, daily_orders or 0,
+        )
+        logger.info("Captured risk snapshot: pnl=%.2f trades=%d", daily_pnl, daily_orders or 0)
+    except Exception:
+        logger.exception("Failed to capture risk snapshot")
+```
+
+**Validates:** Bugfix 2.6, 2.7; Regression 3.10
+
+---
+
+### Fix 5: Prediction Price Fallback (Bug 1.8)
+
+**File:** `services/validation/prediction_snapshot.py`
+
+After the primary `market_snapshots` price lookup returns NULL, add a fallback:
+
+```python
+# After market_snapshots lookup:
+if price_at_prediction is None:
+    pos_row = await conn.fetchrow(
+        "SELECT current_price FROM positions "
+        "WHERE ticker = $1 AND current_price IS NOT NULL LIMIT 1",
+        ticker,
+    )
+    if pos_row:
+        price_at_prediction = float(pos_row["current_price"])
+```
+
+Only covers tickers with open positions (currently 10). Acceptable tradeoff — most active tickers are the ones we hold.
+
+**Validates:** Bugfix 2.8; Regression 3.4
+
+---
+
+### Fix 6: Lake Publisher Scale-Down (Bug 1.9)
+
+**File:** `infra/helm/stonks-oracle/values.yaml`
+
+```yaml
+# Change:
+    replicas: 0
+```
+
+Keeps the deployment definition intact for future use but schedules no pods.
+
+**Validates:** Bugfix 2.9; Regression 3.5
+
+---
+
+### Fix 7: Order rejection_reason Population (Bug 1.10)
+
+**File:** `services/adapters/broker_service.py`
+
+Extend `_INSERT_ORDER` to include `rejection_reason` and `rejected_at`:
+
+```python
+_INSERT_ORDER = """
+INSERT INTO orders (
+    id, recommendation_id, broker_account_id, ticker, side, order_type,
+    quantity, limit_price, stop_price, status, idempotency_key,
+    broker_order_id, decision_trace, submitted_at, filled_at,
+    fill_price, fill_quantity, rejection_reason, rejected_at
+) VALUES (
+    $1::uuid, $2, $3::uuid, $4, $5, $6,
+    $7, $8, $9, $10, $11,
+    $12, $13::jsonb, $14, $15,
+    $16, $17, $18, $19
+)
+ON CONFLICT (idempotency_key) DO UPDATE SET
+    status = EXCLUDED.status,
+    broker_order_id = EXCLUDED.broker_order_id,
+    filled_at = EXCLUDED.filled_at,
+    fill_price = EXCLUDED.fill_price,
+    fill_quantity = EXCLUDED.fill_quantity,
+    rejection_reason = COALESCE(EXCLUDED.rejection_reason, orders.rejection_reason),
+    rejected_at = COALESCE(EXCLUDED.rejected_at, orders.rejected_at),
+    updated_at = NOW()
+"""
+```
+
+Update `persist_order()` to pass the new parameters:
+
+```python
+rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None
+rejected_at = now if resp.status == OrderStatus.REJECTED else None
+# Add as params $18, $19
+```
+
+**Validates:** Bugfix 2.10; Regression 3.6, 3.8
+
+---
+
+## Correctness Properties
+
+Property 1: Rate limit compliance — After fix, the rolling 1-minute window for Polygon requests SHALL NOT exceed the configured limit (default 5). Existing `check_rate_limit()` windowed counter enforces this; we only change the threshold constant.
+
+Property 2: Validation cycle completeness — `prediction_outcomes` row count SHALL grow monotonically after the first validation cycle runs. Each run finds matured snapshots not yet evaluated and persists outcomes.
+
+Property 3: Metric snapshot freshness — `model_metric_snapshots` SHALL contain rows with `generated_at` within the last 2 hours after 2+ validation cycles. The quality gate can then evaluate against real data.
+
+Property 4: Config read correctness — `_read_v3_flag()` SHALL return True when `risk_configs.config->>'v3_engine_enabled'` is `'true'` and False for all other values including NULL or missing key.
+
+Property 5: Snapshot idempotency — `portfolio_snapshots` SHALL contain at most 1 row per `snapshot_date`. The `maybe_capture_daily_snapshots` function checks for existing rows before insert.
+
+Property 6: Rejection reason preservation — Every order with `status = 'rejected'` persisted via `persist_order()` SHALL have a non-NULL `rejection_reason` extracted from the error response.
+
+## Testing Strategy
+
+- **Unit tests:** Update `test_scheduler.py` with a test verifying `run_validation_cycle` is called after the counter threshold. Test `_read_v3_flag` with mocked JSONB config returning various values.
+- **Integration tests:** Verify `persist_order` with rejected status populates `rejection_reason` column.
+- **Manual verification post-deploy:**
+  - `kubectl logs deployment/scheduler -n stonks-beta --tail=100 | grep Validation` shows outcome counts
+  - `SELECT count(*) FROM prediction_outcomes` starts growing within 1 hour
+  - `SELECT count(*) FROM model_metric_snapshots` populates after outcomes exist
+  - Scheduler logs show significantly fewer "Rate limit hit" warnings
+  - Aggregation logs no longer show "column value does not exist" error
+  - After market close: `SELECT * FROM portfolio_snapshots WHERE snapshot_date = CURRENT_DATE` returns 1 row
+
+## Glossary
+
+| Term | Definition |
+|------|-----------|
+| Validation cycle | Hourly scheduler task: evaluate_matured_predictions → compute_and_store_metric_snapshots |
+| Quality gate | Threshold check on model_metric_snapshots that determines if trading can be promoted from paper to live |
+| Prediction snapshot | Frozen state of a recommendation at generation time (prices, evidence, scores) |
+| Outcome evaluation | Matching a matured prediction snapshot against realized market returns |
+| Polygon free tier | API plan with ~5 requests/minute rate limit |

.kiro/specs/ops-pipeline-fixes/tasks.md

@@ -0,0 +1,69 @@
+# Implementation Plan: ops-pipeline-fixes
+
+## Overview
+
+Fix 10 operational bugs preventing the validation/calibration feedback loop from functioning and degrading ingestion throughput. Changes span scheduler (rate limiting + periodic tasks), aggregation worker (config query), broker service (rejection reason), prediction snapshot (price fallback), and Helm chart (dead pod removal).
+
+## Tasks
+
+- [x] 1. Fix Polygon global rate limit — In `services/scheduler/app.py`, replace `POLYGON_GLOBAL_RATE_LIMIT: int = 45` with `POLYGON_GLOBAL_RATE_LIMIT: int = int(os.getenv("POLYGON_GLOBAL_RATE_LIMIT", "5"))` to make it env-configurable and default to the free-tier limit
+    - **Validates: Bugfix 2.1; Regression 3.1, 3.7**
+
+- [x] 2. Fix v3_engine_enabled query — In `services/aggregation/worker.py`, replace `_V3_ENGINE_FLAG_QUERY` from `SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'` to `SELECT config->>'v3_engine_enabled' AS enabled FROM risk_configs WHERE name = 'default' AND active = TRUE LIMIT 1`, and rewrite `_read_v3_flag()` to parse the returned string (checking for "true"/"1"/"yes"), returning False for NULL/missing/error
+    - **Validates: Bugfix 2.2; Regression 3.2**
+
+- [x] 3. Add validation cycle constant and counter — In `services/scheduler/app.py`, add `VALIDATION_CYCLE_INTERVAL = int(os.getenv("VALIDATION_CYCLE_INTERVAL", "240"))` constant and `validation_counter = 0` initialization in `main()`
+    - **Validates: Bugfix 2.3, 2.4**
+
+- [x] 4. Implement run_validation_cycle function — In `services/scheduler/app.py`, implement `run_validation_cycle(pool)` that calls `evaluate_matured_predictions(pool)` followed by `compute_and_store_metric_snapshots(pool)`, with try/except logging for each and skipping metrics if outcomes fail
+    - **Validates: Bugfix 2.3, 2.4, 2.5; Regression 3.3**
+
+- [x] 5. Wire validation cycle into main loop — In `services/scheduler/app.py` main loop, add the counter increment and conditional call to `run_validation_cycle(pool)` after the existing `report_schedule_counter` block
+    - **Validates: Bugfix 2.3, 2.4, 2.5**
+
+- [x] 6. Add snapshot cycle constant and counter — In `services/scheduler/app.py`, add `SNAPSHOT_CYCLE_INTERVAL = int(os.getenv("SNAPSHOT_CYCLE_INTERVAL", "240"))` constant and `snapshot_counter = 0` initialization in `main()`
+    - **Validates: Bugfix 2.6, 2.7**
+
+- [x] 7. Implement maybe_capture_daily_snapshots function — In `services/scheduler/app.py`, implement `maybe_capture_daily_snapshots(pool)` that checks time (after 16:30 ET), checks idempotency (no existing row for today), queries positions table for portfolio value/unrealized PnL, and inserts into `portfolio_snapshots` and `daily_risk_snapshots`
+    - **Validates: Bugfix 2.6, 2.7; Regression 3.10**
+
+- [x] 8. Wire snapshot cycle into main loop — In `services/scheduler/app.py` main loop, add the counter increment and conditional call to `maybe_capture_daily_snapshots(pool)` after the validation counter block
+    - **Validates: Bugfix 2.6, 2.7**
+
+- [x] 9. Add prediction price fallback — In `services/validation/prediction_snapshot.py`, after the primary market_snapshots price lookup returns NULL for `price_at_prediction`, add a fallback query to positions table: `SELECT current_price FROM positions WHERE ticker = $1 AND current_price IS NOT NULL LIMIT 1`
+    - **Validates: Bugfix 2.8; Regression 3.4**
+
+- [x] 10. Scale down lake-publisher — In `infra/helm/stonks-oracle/values.yaml`, change the lake-publisher `replicas` from `1` to `0`
+    - **Validates: Bugfix 2.9; Regression 3.5**
+
+- [x] 11. Extend _INSERT_ORDER SQL — In `services/adapters/broker_service.py`, extend `_INSERT_ORDER` SQL to include `rejection_reason` and `rejected_at` as parameters $18 and $19, with COALESCE in the ON CONFLICT UPDATE clause to preserve existing values
+    - **Validates: Bugfix 2.10; Regression 3.6, 3.8**
+
+- [x] 12. Update persist_order parameters — In `services/adapters/broker_service.py`, update `persist_order()` to compute `rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None` and `rejected_at = now if resp.status == OrderStatus.REJECTED else None`, passing them as the final two parameters in the execute call
+    - **Validates: Bugfix 2.10; Regression 3.6, 3.8**
+
+- [x] 13. Lint and test — Run `.venv/bin/ruff check services/` and `.venv/bin/python -m pytest tests/ -x --tb=short -q` to verify no regressions
+    - **Validates: Regression 3.1–3.10**
+
+## Task Dependency Graph
+
+```json
+{
+  "waves": [
+    {"tasks": [1, 2, 9, 10]},
+    {"tasks": [3, 6, 11]},
+    {"tasks": [4, 7, 12]},
+    {"tasks": [5, 8]},
+    {"tasks": [13]}
+  ]
+}
+```
+
+Tasks 1, 2, 9, 10 are fully independent. Tasks 3/6/11 set up constants needed by 4/7/12. Tasks 5/8 wire into the main loop after their functions exist. Task 13 validates everything last.
+
+## Notes
+
+- No database migrations required — all tables already exist with correct columns
+- All scheduler changes use the existing counter-based periodic task pattern already established for cleanup, aggregation, and report tasks
+- Lazy imports in `run_validation_cycle` avoid circular imports and keep scheduler startup fast
+- The `maybe_capture_daily_snapshots` idempotency check prevents duplicate rows on scheduler restart

.kiro/specs/pipeline-health-fixes/.config.kiro

@@ -0,0 +1 @@
+{"specId": "f5d99301-94ef-4dc2-8ba4-ccefeee7ecba", "workflowType": "requirements-first", "specType": "bugfix"}

.kiro/specs/pipeline-health-fixes/bugfix.md

@@ -0,0 +1,51 @@
+# Bugfix Requirements Document
+
+## Introduction
+
+Five operational bugs in the stonks-beta deployment degrade pipeline health: 1,809 documents stuck in `parsed` status due to recovery batch limits, 26.5% of prediction snapshots missing prices due to incomplete fallback chains, 64% sell bias from uncalibrated NuExtract3 sentiment outputs, idle signal-engine consuming resources while doing nothing, and a quality gate stuck in paper-only mode due to an overly strict staleness threshold interacting with the NULL price problem.
+
+## Bug Analysis
+
+### Current Behavior (Defect)
+
+1.1 WHEN the `recover_stale_documents` task runs with 1,809+ documents stuck in `parsed` status THEN the system only processes 100 per cycle (every ~5 minutes), requiring 90+ cycles (~7.5 hours) to clear the backlog while new documents may continue accumulating
+
+1.2 WHEN a prediction snapshot is created for a ticker without an open position AND without recent market_snapshots data THEN the system stores NULL in `price_at_prediction` because the fallback chain stops at the positions table (26.5% of snapshots affected — 33,324 of 125,590)
+
+1.3 WHEN the outcome evaluator encounters a prediction snapshot with NULL `price_at_prediction` THEN the system skips the snapshot entirely, creating a validation blind spot where 26.5% of predictions are never evaluated
+
+1.4 WHEN the aggregation pipeline processes NuExtract3 extraction outputs THEN the system passes raw `impact_score` and `sentiment` values directly into signal weighting without any distribution normalization, resulting in systematic negative bias producing 64% sell / 23% watch / 12% buy recommendations
+
+1.5 WHEN the signal-engine pod starts with `dual_pipeline_enabled=False` THEN the system enters an infinite sleep loop consuming CPU (100m request / 500m limit) and memory (128Mi request / 256Mi limit) while producing zero signal evaluations
+
+1.6 WHEN the quality gate checks `model_metric_snapshots` freshness with a 24-hour staleness threshold AND the validation cycle skips all predictions due to NULL prices (Bug 1.2/1.3) THEN the system permanently defaults to paper-only mode because no fresh metric snapshots are ever generated
+
+### Expected Behavior (Correct)
+
+2.1 WHEN the scheduler detects more than 100 documents stuck in `parsed` status older than the threshold THEN the system SHALL increase the batch limit for recovery processing (up to 500 per cycle) and provide a one-time management command to bulk-recover the existing backlog without waiting for periodic sweeps
+
+2.2 WHEN a prediction snapshot is created and no price is available from market_snapshots (exact time) or positions table THEN the system SHALL query `market_snapshots` with a wider time window (last 24 hours of bar data for the ticker) as an additional fallback before accepting NULL
+
+2.3 WHEN backfilling existing prediction snapshots with NULL `price_at_prediction` THEN the system SHALL use the extended fallback chain (market_snapshots within 24h of `generated_at`, then positions) to populate prices retroactively via a migration script
+
+2.4 WHEN the aggregation pipeline computes signal weights from impact records THEN the system SHALL apply z-score normalization to `impact_score` values relative to the rolling 7-day distribution of impact records for the same ticker, preventing systematic model bias from dominating the directional signal
+
+2.5 WHEN the signal-engine deployment is not ready for production use (`dual_pipeline_enabled=False`) THEN the system SHALL be scaled to 0 replicas in the Helm values files (beta, paper, live) to eliminate wasted CPU, memory, and any GPU time-slice allocations
+
+2.6 WHEN the quality gate evaluates metric snapshot freshness during the bootstrapping period THEN the system SHALL use a 48-hour staleness threshold (instead of 24h) to tolerate gaps while the validation cycle ramps up after Bug 1.2/1.3 are fixed
+
+### Unchanged Behavior (Regression Prevention)
+
+3.1 WHEN documents enter `parsed` status and are processed within the normal threshold window (< 240 minutes) THEN the system SHALL CONTINUE TO leave them for the extraction queue consumer without interference from the recovery task
+
+3.2 WHEN a prediction snapshot is created and market_snapshots contains a recent bar for the ticker THEN the system SHALL CONTINUE TO use the primary `market_snapshots` close price without invoking any fallback
+
+3.3 WHEN the aggregation pipeline processes tickers with balanced sentiment distributions (equal bullish/bearish evidence) THEN the system SHALL CONTINUE TO produce neutral/mixed recommendations without artificial skew from the normalization step
+
+3.4 WHEN the signal-engine is re-enabled in the future (dual_pipeline_enabled=True with replicas > 0) THEN the system SHALL CONTINUE TO function correctly with its existing queue-based architecture and configuration loading
+
+3.5 WHEN the quality gate evaluates a metric snapshot that is less than 48 hours old and meets all threshold criteria THEN the system SHALL CONTINUE TO promote recommendations to live_eligible mode per existing threshold logic
+
+3.6 WHEN the outcome evaluator processes prediction snapshots with valid (non-NULL) prices THEN the system SHALL CONTINUE TO evaluate them normally and produce prediction_outcomes records
+
+3.7 WHEN the `retry_failed_extractions` task handles documents in `extraction_failed` status THEN the system SHALL CONTINUE TO process them on the existing cadence and logic without interference from the parsed-document recovery changes

.kiro/specs/pipeline-health-fixes/design.md

@@ -0,0 +1,320 @@
+# Pipeline Health Fixes — Bugfix Design
+
+## Overview
+
+Five operational bugs degrade stonks-beta pipeline health. This design formalizes the bug conditions, expected fixes, and validation strategy for each:
+
+1. **Stuck Parsed Docs** — `recover_stale_documents()` batch limit of 100 is too low for 1,809 stuck documents; increase to 500 and lower the stale threshold to 30 minutes.
+2. **Extended Price Fallback** — Prediction snapshots missing prices (26.5%) because the fallback chain stops at `positions`; add a third fallback querying `market_snapshots` within 24h.
+3. **Sentiment Z-Score Normalization** — Raw NuExtract3 `impact_score` values produce 64% sell bias; normalize using 7-day rolling z-scores per ticker before signal weighting.
+4. **Signal Engine Scale Down** — Idle signal-engine pods consume resources; set replicas to 0 in all Helm values files.
+5. **Quality Gate Threshold** — 24h staleness threshold permanently locks quality gate to paper-only; relax to 48h.
+
+## Glossary
+
+- **Bug_Condition (C)**: The specific conditions under which each bug manifests
+- **Property (P)**: The desired correct behavior after the fix is applied
+- **Preservation**: Existing behavior that must remain unchanged after the fix
+- **`recover_stale_documents()`**: Function in `services/scheduler/app.py` that re-enqueues documents stuck in `parsed` status
+- **`STALE_PARSED_THRESHOLD_MINUTES`**: Constant (currently 240) controlling how long a document must be stuck before recovery
+- **`fetch_latest_close_price()`**: Function in `services/validation/prediction_snapshot.py` that queries `market_snapshots` for the most recent bar
+- **`compute_signal_weight()`**: Function in `services/aggregation/scoring.py` that computes combined signal weight from recency, credibility, novelty, confidence, and impact
+- **`QualityGateConfig.max_snapshot_age_hours`**: Threshold in `services/trading/model_quality_gate.py` controlling when the quality gate defaults to paper-only
+
+## Bug Details
+
+### Bug Condition
+
+The pipeline health degradation manifests across five independent conditions:
+
+**Formal Specification:**
+```
+FUNCTION isBugCondition(input)
+  INPUT: input of type PipelineState
+  OUTPUT: boolean
+  
+  -- Bug 1: Parsed docs stuck beyond batch capacity
+  RETURN (input.stuckParsedDocCount > 100 
+          AND input.recoveryBatchLimit == 100
+          AND input.docStaleMinutes >= 240)
+  -- Bug 2: Price fallback chain incomplete
+  OR (input.tickerPrice IS NULL 
+      AND input.positionPrice IS NULL
+      AND input.marketSnapshotWithin24h IS NOT NULL)
+  -- Bug 3: Raw impact scores without normalization
+  OR (input.impactScoreUsedRaw == TRUE
+      AND input.ticker7dStddev > 0)
+  -- Bug 4: Signal engine running idle
+  OR (input.signalEngineReplicas > 0 
+      AND input.dualPipelineEnabled == FALSE)
+  -- Bug 5: Quality gate threshold too strict
+  OR (input.snapshotAgeHours > 24 
+      AND input.snapshotAgeHours <= 48
+      AND input.maxSnapshotAgeConfig == 24)
+END FUNCTION
+```
+
+### Examples
+
+- **Bug 1**: 1,809 documents in `parsed` status older than 4 hours. At 100/cycle every 5 minutes, clearing takes 90+ cycles (~7.5h). With 500/batch, it takes 4 cycles (~20 min).
+- **Bug 2**: Ticker PLTR has no open position and `fetch_latest_close_price` returns NULL, but `market_snapshots` has a bar from 3 hours ago that could serve as price.
+- **Bug 3**: NuExtract3 outputs `impact_score` values clustered around -0.3 to -0.1 for a ticker. Without normalization, `weighted_sentiment_average()` systematically produces negative signals → 64% sell recommendations.
+- **Bug 4**: signal-engine pod starts, detects `dual_pipeline_enabled=False`, enters infinite sleep loop consuming 100m CPU request / 128Mi memory request.
+- **Bug 5**: Quality gate reads `model_metric_snapshots`, finds the most recent is 26h old (because validation skips NULL-price predictions), fails staleness check, forces paper-only mode permanently.
+
+## Expected Behavior
+
+### Preservation Requirements
+
+**Unchanged Behaviors:**
+- Documents entering `parsed` status and processed within the normal threshold window (< 30 min after fix) are left alone for the extraction queue consumer
+- Primary price lookup via `fetch_latest_close_price()` (exact time match from `market_snapshots`) continues as the first-choice price source
+- Tickers with balanced sentiment distributions continue to produce neutral/mixed recommendations without artificial skew
+- Signal-engine's queue-based architecture and configuration loading remain functional when re-enabled with replicas > 0
+- Quality gate threshold logic for snapshots younger than 48h and meeting all criteria continues to promote to `live_eligible`
+- Outcome evaluator continues to process prediction snapshots with valid (non-NULL) prices normally
+- `retry_failed_extractions` task continues on its existing cadence without interference
+
+**Scope:**
+All inputs that do NOT match the bug conditions above should be completely unaffected by these fixes. The fixes are additive (new fallback path, wider batch, normalization layer) or config-only (replica count, threshold constant).
+
+## Hypothesized Root Cause
+
+### Bug 1: Stuck Parsed Docs
+- **Batch limit too small**: The `LIMIT 100` in the SQL query caps recovery throughput at 100 docs per scheduler cycle (~5 min). When a Redis crash orphans thousands of documents, the recovery rate cannot keep up with the backlog.
+- **Threshold too conservative**: `STALE_PARSED_THRESHOLD_MINUTES = 240` (4 hours) means documents must be stuck for 4 hours before recovery kicks in. A 30-minute threshold would catch orphans much faster.
+
+### Bug 2: Incomplete Fallback Chain
+- **Missing time-window query**: `fetch_latest_close_price()` only checks `market_snapshots` for an exact timestamp match. When market data ingestion is delayed or the prediction happens outside market hours, no exact match exists.
+- **Positions-only fallback**: The positions table fallback only works for tickers with an active position. 26.5% of snapshots are for tickers without positions.
+
+### Bug 3: Uncalibrated Impact Scores
+- **No distribution normalization**: NuExtract3 model outputs are passed raw into `compute_signal_weight()` via `impact_score` parameter. The model has a systematic negative bias in its output distribution that is not corrected.
+- **Per-ticker variance ignored**: Different tickers receive different volume/types of news, producing different impact_score distributions. A global normalization would be insufficient.
+
+### Bug 4: Idle Signal Engine
+- **Replicas set to 1 by default**: `values.yaml` defines `signalEngine.replicas: 1` regardless of whether the dual pipeline feature is enabled. The pod starts, detects the feature is off, and sleeps forever.
+
+### Bug 5: Overly Strict Staleness
+- **24h threshold too tight during bootstrapping**: The `max_snapshot_age_hours = 24` default assumes the validation cycle runs frequently. When Bug 2/3 cause most predictions to be skipped, metric snapshots aren't generated, and the 24h window expires.
+
+## Correctness Properties
+
+Property 1: Bug Condition — Stuck Parsed Docs Recovery
+
+_For any_ set of documents stuck in `parsed` status longer than 30 minutes, the fixed `recover_stale_documents()` function SHALL process up to 500 documents per cycle, reducing backlog clearance time by 5x compared to the previous 100-document limit.
+
+**Validates: Requirements 2.1**
+
+Property 2: Bug Condition — Extended Price Fallback
+
+_For any_ prediction snapshot where `fetch_latest_close_price()` returns NULL and the `positions` table has no price, but `market_snapshots` contains a bar for the ticker within 24 hours of the prediction time, the fixed code SHALL use that bar's close price as `price_at_prediction`.
+
+**Validates: Requirements 2.2, 2.3**
+
+Property 3: Bug Condition — Sentiment Z-Score Normalization
+
+_For any_ set of `document_impact_records` for a ticker, the fixed aggregation pipeline SHALL normalize `impact_score` values using the 7-day rolling mean and standard deviation for that ticker before passing them into signal weight computation, preventing systematic model bias.
+
+**Validates: Requirements 2.4**
+
+Property 4: Bug Condition — Signal Engine Scale Down
+
+_For any_ Helm deployment where `dual_pipeline_enabled=False`, the fixed Helm values SHALL specify `signalEngine.replicas: 0`, preventing the pod from being scheduled and consuming resources.
+
+**Validates: Requirements 2.5**
+
+Property 5: Bug Condition — Quality Gate Threshold
+
+_For any_ model metric snapshot that is between 24h and 48h old, the fixed quality gate SHALL NOT reject it as stale, allowing the system to remain in non-paper mode during the bootstrapping period.
+
+**Validates: Requirements 2.6**
+
+Property 6: Preservation — Normal Document Processing
+
+_For any_ document that enters `parsed` status and is processed within 30 minutes, the fixed `recover_stale_documents()` function SHALL NOT interfere with normal extraction queue processing, preserving the existing pipeline flow.
+
+**Validates: Requirements 3.1, 3.7**
+
+Property 7: Preservation — Primary Price Path
+
+_For any_ prediction snapshot where `fetch_latest_close_price()` returns a valid price, the fixed code SHALL use that price directly without invoking any fallback, preserving the primary price lookup behavior.
+
+**Validates: Requirements 3.2**
+
+Property 8: Preservation — Balanced Sentiment
+
+_For any_ ticker with a balanced sentiment distribution (equal bullish/bearish evidence), the z-score normalization SHALL produce values centered around 0, preserving neutral/mixed recommendation output without artificial skew.
+
+**Validates: Requirements 3.3**
+
+Property 9: Preservation — Quality Gate Valid Snapshots
+
+_For any_ model metric snapshot younger than 48h that meets all threshold criteria, the fixed quality gate SHALL continue to promote recommendations to `live_eligible` mode per existing logic.
+
+**Validates: Requirements 3.5, 3.6**
+
+## Fix Implementation
+
+### Changes Required
+
+**Bug 1: Stuck Parsed Docs Recovery**
+
+**File**: `services/scheduler/app.py`
+
+**Function**: `recover_stale_documents()`
+
+**Specific Changes**:
+1. **Lower stale threshold**: Change `STALE_PARSED_THRESHOLD_MINUTES` from `240` to `30` — documents stuck longer than 30 minutes are likely orphaned
+2. **Increase batch limit**: Change `LIMIT 100` to `LIMIT 500` in the SQL query
+3. **Update enqueued TTL**: Change `_ENQUEUED_TTL` from `14400` (4h) to `3600` (1h) to match the new threshold
+
+---
+
+**Bug 2: Extended Price Fallback**
+
+**File**: `services/validation/prediction_snapshot.py`
+
+**Function**: `create_prediction_snapshot()`
+
+**Specific Changes**:
+1. **Add market_snapshots time-window fallback**: After the positions fallback fails, query `market_snapshots` for the most recent bar within 24h of the current time for the ticker
+2. **SQL query**: `SELECT close FROM market_snapshots WHERE ticker = $1 AND timestamp >= NOW() - INTERVAL '24 hours' ORDER BY timestamp DESC LIMIT 1`
+3. **Log the fallback**: Add info-level logging when the extended fallback is used
+
+**New File**: `scripts/backfill_snapshot_prices.py`
+
+**Purpose**: One-time backfill script to populate `price_at_prediction` for existing NULL snapshots using the extended fallback chain.
+
+**Approach**:
+1. Query all `prediction_snapshots` where `price_at_prediction IS NULL`
+2. For each, attempt: `market_snapshots` within 24h of `generated_at`, then `positions` table
+3. Update the row with the found price
+4. Report statistics (found via market_snapshots, found via positions, still NULL)
+
+---
+
+**Bug 3: Sentiment Z-Score Normalization**
+
+**File**: `services/aggregation/worker.py` (or new helper in `services/aggregation/scoring.py`)
+
+**Function**: New function `normalize_impact_scores()` called before `compute_signal_weight()`
+
+**Specific Changes**:
+1. **Add normalization function**: Compute 7-day rolling mean and stddev of `impact_score` per ticker from `document_impact_records`
+2. **Formula**: `normalized = (raw - mean_7d) / max(stddev_7d, 0.1)` — the 0.1 floor prevents division by near-zero stddev for low-activity tickers
+3. **Integration point**: In the aggregation loop (around line 440 of worker.py), normalize `imp.impact_score` before passing to `compute_signal_weight()` and `WeightedSignal`
+4. **Fallback**: If fewer than 5 records exist in the 7-day window, use the raw score (insufficient data for meaningful normalization)
+5. **Query**: `SELECT AVG(impact_score) as mean, STDDEV(impact_score) as stddev FROM document_impact_records WHERE ticker = $1 AND created_at >= NOW() - INTERVAL '7 days'`
+
+---
+
+**Bug 4: Signal Engine Scale Down**
+
+**Files**: `infra/helm/stonks-oracle/values.yaml`, `values-beta.yaml`, `values-paper.yaml`
+
+**Specific Changes**:
+1. **values.yaml**: Change `signalEngine.replicas` from `1` to `0`
+2. **values-beta.yaml**: Add `signalEngine.replicas: 0` under `services:`
+3. **values-paper.yaml**: Add `signalEngine.replicas: 0` under `services:`
+
+---
+
+**Bug 5: Quality Gate Threshold**
+
+**File**: `services/trading/model_quality_gate.py`
+
+**Class**: `QualityGateConfig`
+
+**Specific Changes**:
+1. **Change default**: `max_snapshot_age_hours: int = 48` (was 24)
+
+## Testing Strategy
+
+### Validation Approach
+
+The testing strategy follows a two-phase approach: first, surface counterexamples that demonstrate the bug on unfixed code, then verify the fix works correctly and preserves existing behavior.
+
+### Exploratory Bug Condition Checking
+
+**Goal**: Surface counterexamples that demonstrate the bugs BEFORE implementing the fixes. Confirm or refute the root cause analysis.
+
+**Test Plan**: Write tests that exercise each bug condition on the unfixed code to observe failures.
+
+**Test Cases**:
+1. **Batch Overflow Test**: Create 600 documents in `parsed` status older than threshold, run `recover_stale_documents()`, assert only 100 are processed (will demonstrate Bug 1)
+2. **Price Fallback Gap Test**: Call `create_prediction_snapshot()` for a ticker with no position and no exact market_snapshots match, assert `price_at_prediction` is NULL (will demonstrate Bug 2)
+3. **Sentiment Bias Test**: Generate 50 impact records with systematic negative bias (mean=-0.3, stddev=0.1), compute weighted signals, assert directional signal is negative (will demonstrate Bug 3)
+4. **Quality Gate Staleness Test**: Set most recent metric snapshot to 26h ago, evaluate quality gate, assert it fails (will demonstrate Bug 5)
+
+**Expected Counterexamples**:
+- Bug 1: Only 100 of 600 documents recovered per cycle
+- Bug 2: `price_at_prediction` stored as NULL despite market data existing within 24h
+- Bug 3: Weighted sentiment average heavily negative despite mixed underlying events
+- Bug 5: Quality gate returns `passed=False` with reason containing "stale"
+
+### Fix Checking
+
+**Goal**: Verify that for all inputs where the bug condition holds, the fixed function produces the expected behavior.
+
+**Pseudocode:**
+```
+FOR ALL input WHERE isBugCondition(input) DO
+  result := fixedFunction(input)
+  ASSERT expectedBehavior(result)
+END FOR
+```
+
+**Per-bug fix checks:**
+- Bug 1: `recover_stale_documents()` processes up to 500 docs with 30-min threshold
+- Bug 2: Extended fallback returns a price when `market_snapshots` has data within 24h
+- Bug 3: Normalized impact scores have mean ≈ 0 and stddev ≈ 1 for active tickers
+- Bug 4: `kubectl get pods` shows 0 signal-engine pods
+- Bug 5: Quality gate passes for snapshots 24–48h old that meet metric thresholds
+
+### Preservation Checking
+
+**Goal**: Verify that for all inputs where the bug condition does NOT hold, the fixed function produces the same result as the original function.
+
+**Pseudocode:**
+```
+FOR ALL input WHERE NOT isBugCondition(input) DO
+  ASSERT originalFunction(input) = fixedFunction(input)
+END FOR
+```
+
+**Testing Approach**: Property-based testing is recommended for preservation checking because:
+- It generates many test cases automatically across the input domain
+- It catches edge cases that manual unit tests might miss
+- It provides strong guarantees that behavior is unchanged for all non-buggy inputs
+
+**Test Plan**: Observe behavior on UNFIXED code first for normal inputs, then write property-based tests capturing that behavior.
+
+**Test Cases**:
+1. **Normal Doc Processing Preservation**: Documents < 30 min old are never touched by recovery
+2. **Primary Price Preservation**: When `fetch_latest_close_price()` succeeds, no fallback is invoked
+3. **Balanced Sentiment Preservation**: Tickers with symmetric impact_score distributions produce neutral signals after normalization
+4. **Quality Gate Normal Preservation**: Snapshots < 48h old and meeting thresholds still pass
+5. **Failed Extraction Preservation**: `retry_failed_extractions()` behavior unchanged
+
+### Unit Tests
+
+- Test `recover_stale_documents()` with various document counts (0, 50, 500, 1000)
+- Test extended price fallback with market_snapshots at various time offsets (1h, 12h, 23h, 25h)
+- Test z-score normalization with known distributions (mean=0, mean=-0.5, stddev=0, stddev=0.05)
+- Test quality gate with snapshot ages at boundary (23h, 24h, 47h, 48h, 49h)
+- Test backfill script with mixed NULL/non-NULL snapshots
+
+### Property-Based Tests
+
+- Generate random document ages and counts, verify recovery processes correct subset (> 30 min old, up to 500)
+- Generate random ticker price scenarios, verify fallback chain ordering is preserved (primary → positions → market_snapshots_24h → NULL)
+- Generate random impact_score distributions per ticker, verify normalized output has bounded variance and zero-centered mean
+- Generate random snapshot ages, verify quality gate accepts [0, 48h) and rejects [48h, ∞)
+
+### Integration Tests
+
+- End-to-end: create documents in `parsed` status, run scheduler cycle, verify extraction queue populated
+- End-to-end: create prediction snapshot for ticker without position, verify price populated from market_snapshots
+- End-to-end: run full aggregation cycle with biased NuExtract3 outputs, verify recommendation direction is not systematically biased
+- Helm template render: verify signal-engine deployment has 0 replicas in all value files

.kiro/specs/pipeline-health-fixes/tasks.md

@@ -0,0 +1,174 @@
+# Implementation Plan
+
+## Overview
+
+Bugfix implementation for five pipeline health issues: stuck parsed docs, missing price fallback, uncalibrated sentiment scores, idle signal-engine pods, and overly strict quality gate threshold. Tasks follow the exploratory bugfix workflow: explore bugs via tests, preserve existing behavior, implement fixes, validate.
+
+## Tasks
+
+- [x] 1. Write bug condition exploration test
+  - **Property 1: Bug Condition** - Pipeline Health Degradation
+  - **CRITICAL**: This test MUST FAIL on unfixed code - failure confirms the bugs exist
+  - **DO NOT attempt to fix the test or the code when it fails**
+  - **NOTE**: This test encodes the expected behavior - it will validate the fix when it passes after implementation
+  - **GOAL**: Surface counterexamples that demonstrate all five bugs exist
+  - **Scoped PBT Approach**: Scope properties to the concrete failing cases for each bug condition
+  - Test file: `tests/test_pbt_pipeline_health_bug_condition.py`
+  - **Bug 1 - Batch Overflow**: Create 600 documents in `parsed` status older than 30 min, run `recover_stale_documents()`, assert up to 500 are recovered per cycle (will FAIL on unfixed code which caps at 100)
+  - **Bug 2 - Price Fallback Gap**: Call `create_prediction_snapshot()` for ticker with no position and no exact market_snapshots match but data within 24h exists, assert `price_at_prediction` is NOT NULL (will FAIL on unfixed code which returns NULL)
+  - **Bug 3 - Sentiment Bias**: Generate 50 impact records with systematic negative bias (mean=-0.3, stddev=0.1), compute weighted signals via aggregation, assert normalized output is zero-centered (will FAIL on unfixed code which passes raw scores)
+  - **Bug 5 - Quality Gate Staleness**: Set most recent metric snapshot to 26h ago, evaluate quality gate, assert it passes (will FAIL on unfixed code which rejects at 24h)
+  - Run tests on UNFIXED code
+  - **EXPECTED OUTCOME**: Tests FAIL (this is correct - it proves the bugs exist)
+  - Document counterexamples: batch capped at 100, price stored as NULL, sentiment heavily negative, quality gate returns `passed=False`
+  - Mark task complete when tests are written, run, and failures are documented
+  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.6_
+
+- [x] 2. Write preservation property tests (BEFORE implementing fix)
+  - **Property 2: Preservation** - Pipeline Behavior Unchanged for Non-Bug Inputs
+  - **IMPORTANT**: Follow observation-first methodology
+  - Test file: `tests/test_pbt_pipeline_health_preservation.py`
+  - **Normal Doc Processing**: Observe that documents < 30 min old are never touched by `recover_stale_documents()` on unfixed code. Write property: for all documents with age < 30 min, recovery task does NOT enqueue them.
+  - **Primary Price Path**: Observe that when `fetch_latest_close_price()` returns a valid price, no fallback is invoked. Write property: for all tickers where primary price exists, result equals primary price.
+  - **Balanced Sentiment**: Observe that tickers with symmetric impact_score distributions (mean ≈ 0) produce neutral signals. Write property: for all impact_score sets with mean ≈ 0, normalized output remains centered around 0.
+  - **Quality Gate Normal**: Observe that snapshots < 48h old meeting thresholds pass the quality gate. Write property: for all snapshot ages in [0, 48h) meeting metric criteria, quality gate returns `passed=True`.
+  - **Failed Extraction Independence**: Observe `retry_failed_extractions()` behavior is unaffected. Write property: for all documents in `extraction_failed` status, retry logic unchanged.
+  - Run tests on UNFIXED code
+  - **EXPECTED OUTCOME**: Tests PASS (this confirms baseline behavior to preserve)
+  - Mark task complete when tests are written, run, and passing on unfixed code
+  - _Requirements: 3.1, 3.2, 3.3, 3.5, 3.6, 3.7_
+
+- [x] 3. Fix: Signal Engine Scale Down (Helm values)
+
+  - [x] 3.1 Set signal-engine replicas to 0 in all Helm values files
+    - In `infra/helm/stonks-oracle/values.yaml`: change `signalEngine.replicas` from `1` to `0`
+    - In `infra/helm/stonks-oracle/values-beta.yaml`: add/set `signalEngine.replicas: 0` under `services:`
+    - In `infra/helm/stonks-oracle/values-paper.yaml`: add/set `signalEngine.replicas: 0` under `services:`
+    - _Bug_Condition: input.signalEngineReplicas > 0 AND input.dualPipelineEnabled == FALSE_
+    - _Expected_Behavior: signalEngine.replicas == 0 when dual pipeline disabled_
+    - _Preservation: Signal-engine architecture remains functional when re-enabled with replicas > 0_
+    - _Requirements: 2.5, 3.4_
+
+- [x] 4. Fix: Quality Gate Threshold Relaxation
+
+  - [x] 4.1 Change max_snapshot_age_hours default from 24 to 48
+    - File: `services/trading/model_quality_gate.py`
+    - In `QualityGateConfig` class, change `max_snapshot_age_hours: int = 24` to `max_snapshot_age_hours: int = 48`
+    - _Bug_Condition: input.snapshotAgeHours > 24 AND input.snapshotAgeHours <= 48 AND input.maxSnapshotAgeConfig == 24_
+    - _Expected_Behavior: Quality gate accepts snapshots up to 48h old_
+    - _Preservation: Snapshots < 48h meeting criteria continue to promote to live_eligible_
+    - _Requirements: 2.6, 3.5_
+
+- [x] 5. Fix: Stuck Parsed Docs Recovery
+
+  - [x] 5.1 Lower STALE_PARSED_THRESHOLD_MINUTES from 240 to 30
+    - File: `services/scheduler/app.py`
+    - Change constant: `STALE_PARSED_THRESHOLD_MINUTES = 30`
+    - Documents stuck longer than 30 minutes are likely orphaned
+    - _Requirements: 2.1_
+
+  - [x] 5.2 Increase recovery batch LIMIT from 100 to 500
+    - File: `services/scheduler/app.py`
+    - In `recover_stale_documents()` SQL query, change `LIMIT 100` to `LIMIT 500`
+    - _Requirements: 2.1_
+
+  - [x] 5.3 Update _ENQUEUED_TTL from 14400 to 3600
+    - File: `services/scheduler/app.py`
+    - Change `_ENQUEUED_TTL = 3600` (1 hour, matching the new recovery cadence)
+    - _Bug_Condition: input.stuckParsedDocCount > 100 AND input.recoveryBatchLimit == 100 AND input.docStaleMinutes >= 240_
+    - _Expected_Behavior: Recovery processes up to 500 docs per cycle with 30-min threshold_
+    - _Preservation: Documents < 30 min old left alone for extraction queue consumer_
+    - _Requirements: 2.1, 3.1, 3.7_
+
+- [x] 6. Fix: Extended Price Fallback
+
+  - [x] 6.1 Add market_snapshots 24h time-window fallback to create_prediction_snapshot()
+    - File: `services/validation/prediction_snapshot.py`
+    - After positions fallback fails, query: `SELECT close FROM market_snapshots WHERE ticker = $1 AND timestamp >= NOW() - INTERVAL '24 hours' ORDER BY timestamp DESC LIMIT 1`
+    - Add info-level logging when extended fallback is used
+    - _Bug_Condition: input.tickerPrice IS NULL AND input.positionPrice IS NULL AND input.marketSnapshotWithin24h IS NOT NULL_
+    - _Expected_Behavior: Use market_snapshots bar close price as price_at_prediction_
+    - _Preservation: Primary fetch_latest_close_price() path unchanged when it returns a valid price_
+    - _Requirements: 2.2, 3.2_
+
+  - [x] 6.2 Create backfill script scripts/backfill_snapshot_prices.py
+    - Query all `prediction_snapshots` where `price_at_prediction IS NULL`
+    - For each, attempt: `market_snapshots` within 24h of `generated_at`, then `positions` table
+    - Update row with found price
+    - Report statistics: found via market_snapshots, found via positions, still NULL
+    - _Requirements: 2.3_
+
+- [x] 7. Fix: Sentiment Z-Score Normalization
+
+  - [x] 7.1 Add normalize_impact_scores() function
+    - File: `services/aggregation/scoring.py` (new helper function)
+    - Query 7-day mean and stddev per ticker: `SELECT AVG(impact_score) as mean, STDDEV(impact_score) as stddev FROM document_impact_records WHERE ticker = $1 AND created_at >= NOW() - INTERVAL '7 days'`
+    - Formula: `normalized = (raw - mean_7d) / max(stddev_7d, 0.1)`
+    - Fallback: if fewer than 5 records in 7-day window, return raw score unchanged
+    - The 0.1 floor prevents division by near-zero stddev for low-activity tickers
+    - _Requirements: 2.4_
+
+  - [x] 7.2 Integrate normalization into aggregation loop
+    - File: `services/aggregation/worker.py`
+    - Before `compute_signal_weight()` call (around line 440), normalize `imp.impact_score` via `normalize_impact_scores()`
+    - Pass normalized value into `compute_signal_weight()` and `WeightedSignal`
+    - _Bug_Condition: input.impactScoreUsedRaw == TRUE AND input.ticker7dStddev > 0_
+    - _Expected_Behavior: Normalized impact scores with mean ≈ 0, stddev ≈ 1 for active tickers_
+    - _Preservation: Tickers with balanced distributions continue to produce neutral signals_
+    - _Requirements: 2.4, 3.3_
+
+- [x] 8. Verify fixes pass all tests
+
+  - [x] 8.1 Verify bug condition exploration test now passes
+    - **Property 1: Expected Behavior** - Pipeline Health Bugs Resolved
+    - **IMPORTANT**: Re-run the SAME test from task 1 - do NOT write a new test
+    - The test from task 1 encodes the expected behavior for all five bugs
+    - Run `tests/test_pbt_pipeline_health_bug_condition.py`
+    - **EXPECTED OUTCOME**: Test PASSES (confirms bugs are fixed)
+    - _Requirements: 2.1, 2.2, 2.4, 2.6_
+
+  - [x] 8.2 Verify preservation tests still pass
+    - **Property 2: Preservation** - Pipeline Behavior Unchanged for Non-Bug Inputs
+    - **IMPORTANT**: Re-run the SAME tests from task 2 - do NOT write new tests
+    - Run `tests/test_pbt_pipeline_health_preservation.py`
+    - **EXPECTED OUTCOME**: Tests PASS (confirms no regressions)
+    - Confirm all preservation properties still hold after fixes
+    - _Requirements: 3.1, 3.2, 3.3, 3.5, 3.6, 3.7_
+
+- [x] 9. Lint and final validation
+  - Run `.venv/bin/ruff check services/` and fix any lint errors
+  - Run `.venv/bin/python -m pytest tests/ -x --tb=short -q` to confirm full test suite passes
+  - Verify Helm template renders correctly with 0 signal-engine replicas
+  - _Requirements: all_
+
+- [x] 10. Checkpoint - Ensure all tests pass
+  - Ensure all tests pass, ask the user if questions arise.
+
+## Task Dependency Graph
+
+```json
+{
+  "waves": [
+    {"tasks": ["1", "2"]},
+    {"tasks": ["3", "4"]},
+    {"tasks": ["5"]},
+    {"tasks": ["6"]},
+    {"tasks": ["7"]},
+    {"tasks": ["8"]},
+    {"tasks": ["9"]},
+    {"tasks": ["10"]}
+  ]
+}
+```
+
+Tasks 3, 4 are independent config changes (no code deps).
+Task 5 is independent but task 6 builds on the price fallback concept.
+Task 7 is the most complex (new function + integration).
+Tasks 8-10 must run after all fixes are applied.
+
+## Notes
+
+- Bug 4 (signal engine) is validated by Helm template rendering, not a unit test
+- The backfill script (6.2) is a one-time operation, not covered by recurring tests
+- Preservation tests use Hypothesis with `@settings(max_examples=100)` per project conventions
+- Test files follow `test_pbt_*` naming convention per project standards

.kiro/steering/development-process.md

@@ -35,12 +35,22 @@
 - Push to Gitea: `git push gitea main`
 - Gitea remote: `http://admin:<password>@10.1.1.12:30300/admin/stonks-oracle.git`
 - Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
+- Build pipelines split across `build-1.yml`, `build-2.yml`, `build-3.yml` for parallelism
 - ArgoCD watches Gitea `main` and auto-syncs beta/paper/live stages
 - **Do NOT push directly to GitHub** — GitHub is the promotion target after CI passes
 - Once Woodpecker builds and tests pass, code is promoted to GitHub (`git push origin main`)
 - CI handles all image builds and pushes — do NOT manually docker push
 - Check Woodpecker CI status from the Gitea web UI or Woodpecker dashboard
 
+### Dashboard Build (npm ci in K8s)
+- `build-3.yml` has a `npm-install-dashboard` step that runs `npm ci` in a `node:24-alpine` pod
+- K8s CoreDNS causes `EAI_AGAIN` errors for Node.js under concurrent DNS lookups
+- Fix: the step resolves `registry.npmjs.org` to IPv4 via Google DoH and pins it in `/etc/hosts`
+- `NODE_OPTIONS=--dns-result-order=ipv4first` env var is set as additional safety
+- `frontend/.dockerignore` must NOT exclude `node_modules` — the Dockerfile expects it pre-staged
+- The subsequent `build-dashboard` step uses `frontend/` as Docker context (includes `node_modules`)
+- If `npm ci` hangs: check `/etc/hosts` pinning worked, check `npm config set loglevel http` for which request is stuck
+
 ## Deploy
 - Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1)
 - Full teardown: `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1)
@@ -93,6 +103,11 @@ Ingestion jobs MUST include `source_id`, `source_type`, `ticker`, `company_id`,
 - The `competitor_relationships` table uses UUID company IDs — queries must join through `companies` to match by ticker
 - The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI
 - Ingestion jobs require `source_id` from the `sources` table — don't just pass `ticker`
+- `frontend/.dockerignore` must NOT contain `node_modules` — the CI pre-installs it and the Dockerfile relies on `COPY . .` including it
+- `npm config set prefer-ip-address-family 4` does NOT exist in npm 10.x (Node 24) — don't use it
+- Woodpecker `environment:` uses map syntax (`KEY: "value"`) not list syntax (`- KEY=value`)
+- Node.js in Alpine K8s pods gets `EAI_AGAIN` from CoreDNS under load — pin hostnames in `/etc/hosts` for reliability
+- Every Helm-deployed service MUST have a corresponding image build step in `.woodpecker/build-*.yml`
 - **Bash `!` in passwords/strings**: Bash interprets `!` inside double quotes as history expansion. NEVER use double quotes around strings containing `!`. Use single quotes instead: `'St0nks0racl3!'`. For kubectl exec with psql, use: `kubectl exec ... -- psql -U postgres -c "ALTER USER x WITH PASSWORD '"'"'password!'"'"';"` (single-quote escaping trick)
 
 ## No Premature Simplification

.kiro/steering/project-context.md

@@ -46,12 +46,20 @@ Three-layer signal aggregation engine:
 - Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
 - Images tagged as `registry.celestium.life/stonks-oracle/<service>:<sha>` and `:latest`
 - Dashboard image: `frontend/Dockerfile` (multi-stage: node:24 → nginx-unprivileged on port 8080)
+- Dashboard build: `npm-install-dashboard` step in `build-3.yml` pre-installs `node_modules`, then `build-dashboard` runs the Docker build with `node_modules` in context
 - Superset image: `docker/Dockerfile.superset` (apache/superset + trino + psycopg2)
 - Python service images: `docker/Dockerfile` with `SERVICE_CMD` build arg
+- Specialist image: built in `build-3.yml` like other Python services (`SERVICE_CMD=uvicorn services.specialist.app:app --host 0.0.0.0 --port 8000`)
 - Let CI handle image builds and pushes — do NOT manually `docker build && docker push`
 - **Do NOT push directly to GitHub** — GitHub (`origin`) is the promotion target after CI builds and tests pass
 - Promotion to GitHub: `git push origin main` (only after Woodpecker CI succeeds)
 
+### CI DNS Workaround (npm)
+- K8s CoreDNS causes `EAI_AGAIN` (temporary DNS failure) for Node.js/libuv under concurrent requests
+- Fix: `npm-install-dashboard` step resolves `registry.npmjs.org` via Google DoH (`dns.google/resolve`) and pins the IPv4 address in `/etc/hosts` before running `npm ci`
+- `NODE_OPTIONS=--dns-result-order=ipv4first` is set as env var (belt-and-suspenders)
+- `frontend/.dockerignore` does NOT exclude `node_modules` (it must be in the Docker build context since the Dockerfile has no `RUN npm ci`)
+
 ## Deployment Scripts
 - `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available)
 - `~/sources/kube/stonks-oracle/runmelast.sh` — teardown: Helm uninstall, clean resources (preserves DB/MinIO/Redis)

.woodpecker/build-1.yml

@@ -3,6 +3,11 @@ depends_on:
 when:
   event: push
   branch: main
+clone:
+  git:
+    image: woodpeckerci/plugin-git
+    settings:
+      remote: http://10.43.73.77:3000/admin/stonks-oracle.git
 steps:
   build-scheduler:
     image: woodpeckerci/plugin-docker-buildx
@@ -14,7 +19,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -49,7 +60,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -88,7 +105,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -127,7 +150,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""

.woodpecker/build-2.yml

@@ -3,6 +3,11 @@ depends_on:
 when:
   event: push
   branch: main
+clone:
+  git:
+    image: woodpeckerci/plugin-git
+    settings:
+      remote: http://10.43.73.77:3000/admin/stonks-oracle.git
 steps:
   build-extractor:
     image: woodpeckerci/plugin-docker-buildx
@@ -14,7 +19,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -53,7 +64,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -92,7 +109,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -131,7 +154,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -170,7 +199,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""

.woodpecker/build-3.yml

@@ -3,6 +3,11 @@ depends_on:
 when:
   event: push
   branch: main
+clone:
+  git:
+    image: woodpeckerci/plugin-git
+    settings:
+      remote: http://10.43.73.77:3000/admin/stonks-oracle.git
 steps:
   build-broker-adapter:
     image: woodpeckerci/plugin-docker-buildx
@@ -14,7 +19,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -53,7 +64,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -92,7 +109,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -131,7 +154,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -160,6 +189,76 @@ steps:
             memory: 2Gi
             cpu: 4000m
     depends_on: []
+  build-specialist:
+    image: woodpeckerci/plugin-docker-buildx
+    privileged: true
+    settings:
+      repo: registry.celestium.life/stonks-oracle/specialist
+      registry: registry.celestium.life
+      buildx_image: registry.celestium.life/dockerhub-cache/moby/buildkit:buildx-stable-1
+      add_host: registry.celestium.life:10.1.1.12
+      buildx_flags: --driver-opt network=host
+      insecure: true
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
+      http_proxy: ""
+      https_proxy: ""
+      no_proxy: ""
+      logins:
+      - registry: https://registry.celestium.life
+        username:
+          from_secret: harbor_username
+        password:
+          from_secret: harbor_password
+      tags:
+      - ${CI_COMMIT_SHA}
+      - latest
+      dockerfile: docker/Dockerfile
+      no_cache: true
+      context: .
+      build_args:
+      - CACHE_BUST=${CI_COMMIT_SHA}
+      - SERVICE_CMD=uvicorn services.specialist.app:app --host 0.0.0.0 --port 8000
+    backend_options:
+      kubernetes:
+        resources:
+          requests:
+            memory: 1Gi
+            cpu: 1000m
+          limits:
+            memory: 2Gi
+            cpu: 4000m
+    depends_on: []
+  npm-install-dashboard:
+    image: registry.celestium.life/dockerhub-cache/library/node:24-alpine
+    environment:
+      NODE_OPTIONS: "--dns-result-order=ipv4first"
+    commands:
+      - echo "=== Pinning registry.npmjs.org to IPv4 in /etc/hosts ==="
+      - REGISTRY_IP=$(wget -4 -q -O- https://dns.google/resolve?name=registry.npmjs.org\&type=A 2>/dev/null | sed -n 's/.*"data":"\([0-9.]*\)".*/\1/p' | head -1)
+      - echo "Resolved registry.npmjs.org to $REGISTRY_IP"
+      - if [ -n "$REGISTRY_IP" ]; then echo "$REGISTRY_IP registry.npmjs.org" >> /etc/hosts; else echo "104.16.1.35 registry.npmjs.org" >> /etc/hosts; fi
+      - cat /etc/hosts
+      - echo "=== npm/node versions ==="
+      - node --version
+      - npm --version
+      - echo "=== Starting npm ci ==="
+      - cd frontend && npm ci
+    backend_options:
+      kubernetes:
+        resources:
+          requests:
+            memory: 1Gi
+            cpu: 1000m
+          limits:
+            memory: 2Gi
+            cpu: 4000m
+    depends_on: []
   build-dashboard:
     image: woodpeckerci/plugin-docker-buildx
     privileged: true
@@ -170,7 +269,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""
@@ -195,7 +300,8 @@ steps:
           limits:
             memory: 2Gi
             cpu: 4000m
-    depends_on: []
+    depends_on:
+      - npm-install-dashboard
   build-superset:
     image: woodpeckerci/plugin-docker-buildx
     privileged: true
@@ -206,7 +312,13 @@ steps:
       add_host: registry.celestium.life:10.1.1.12
       buildx_flags: --driver-opt network=host
       insecure: true
-      buildkitd_config: "[registry.\"registry.celestium.life\"]\n  http = true\n  insecure = true\n[registry.\"docker.io\"]\n  mirrors = [\"registry.celestium.life/v2/dockerhub-cache\"]\n[registry.\"ghcr.io\"]\n  mirrors = [\"registry.celestium.life/v2/ghcr-cache\"]\n"
+      buildkit_config: |
+        [registry."registry.celestium.life"]
+          insecure = true
+        [registry."docker.io"]
+          mirrors = ["registry.celestium.life/v2/dockerhub-cache"]
+        [registry."ghcr.io"]
+          mirrors = ["registry.celestium.life/v2/ghcr-cache"]
       http_proxy: ""
       https_proxy: ""
       no_proxy: ""

.woodpecker/finalize.yml

@@ -8,6 +8,12 @@ when:
   event: push
   branch: main
 
+clone:
+  git:
+    image: woodpeckerci/plugin-git
+    settings:
+      remote: http://10.43.73.77:3000/admin/stonks-oracle.git
+
 steps:
   integration-test:
     image: registry.celestium.life/dockerhub-cache/alpine/k8s:1.30.2

.woodpecker/test.yml

@@ -2,6 +2,11 @@ when:
   event:
   - push
   - pull_request
+clone:
+  git:
+    image: woodpeckerci/plugin-git
+    settings:
+      remote: http://10.43.73.77:3000/admin/stonks-oracle.git
 steps:
   lint-python:
     image: registry.celestium.life/dockerhub-cache/library/python:3.12-slim

docs/intelligence-pipeline-v3/annotation-guidelines.md

@@ -0,0 +1,363 @@
+# V3 Annotation Guidelines
+
+**Schema version:** 1.0.0
+**Last updated:** 2025-01-15
+
+## Purpose
+
+These guidelines define how human annotators and automated systems label documents in the Intelligence Pipeline v3 Gold Corpus. Every annotation must be evidence-grounded — no label is valid without a supporting evidence span traceable to the source text.
+
+## Core Principles
+
+1. **Evidence first.** If you cannot point to exact text that supports a label, do not apply the label.
+2. **Explicit over inferred.** Mark only what the document explicitly states in primary annotations. Inferred exposure uses a separate, lower-confidence channel.
+3. **Precision over recall.** A missed entity is preferable to a fabricated one. The pipeline uses multiple stages — later stages catch omissions.
+4. **Reproducibility.** Two annotators given the same document should produce substantially the same labels. Ambiguous cases are marked, not resolved by guess.
+
+---
+
+## Evidence Spans
+
+### Definition
+
+An evidence span is the exact substring of the source document that supports an annotation. It uses zero-based character offsets into the original (pre-chunking) document text.
+
+### Rules
+
+- Every entity, event, relation, numeric fact, and sentiment annotation MUST reference at least one evidence span.
+- Spans should be minimal but complete — include enough context for the label to be verifiable without the full document.
+- Overlapping spans are permitted (e.g., the same sentence supports both an entity and an event).
+- The `text` field MUST exactly match `source_text[start_char:end_char]`.
+
+### Positive example
+
+```
+Source: "Apple Inc. reported quarterly earnings of $1.52 per share"
+Span:   start_char=0, end_char=10, text="Apple Inc."
+```
+
+### Negative example
+
+```
+Source: "Apple Inc. reported quarterly earnings of $1.52 per share"
+Span:   start_char=0, end_char=5, text="Apple"
+```
+❌ Truncating "Apple Inc." to "Apple" loses the corporate suffix needed to distinguish from Apple Records or the fruit.
+
+---
+
+## Entity Annotation
+
+### Entity Types
+
+| Type | When to use | Example |
+|------|-------------|---------|
+| `company` | Legal entity, publicly traded firm, government agency | "Apple Inc.", "The Federal Reserve" |
+| `person` | Named individual | "Tim Cook", "Jerome Powell" |
+| `product` | Named product or service | "iPhone 16", "Azure OpenAI Service" |
+| `event` | Named event instance | "Q1 2025 earnings call" |
+| `financial_metric` | Named metric class | "EPS", "revenue", "free cash flow" |
+| `date` | Temporal expression | "Q1 2025", "January 15, 2025" |
+| `percentage` | Percentage value | "4%", "25 basis points" |
+| `currency` | Monetary value | "$1.52", "$10 billion" |
+| `relationship` | Explicit relationship mention | "subsidiary", "joint venture partner" |
+
+### Canonical Resolution
+
+- If an entity maps to a company in the symbol registry, set `canonical_id` and `canonical_name` (ticker).
+- If an entity is ambiguous (e.g., "Apple" could be AAPL or a fruit company), mark an ambiguity marker and set confidence below 1.0.
+- Do NOT invent canonical IDs. If not in the registry, leave `canonical_id` as null.
+
+### Positive example
+
+```json
+{
+  "entity_type": "company",
+  "literal_text": "Alphabet",
+  "canonical_id": "googl-uuid",
+  "canonical_name": "GOOGL",
+  "confidence": 0.97
+}
+```
+
+### Negative example
+
+```json
+{
+  "entity_type": "company",
+  "literal_text": "the company",
+  "canonical_id": "aapl-uuid",
+  "canonical_name": "AAPL",
+  "confidence": 0.90
+}
+```
+❌ "the company" is a pronoun reference, not an entity mention. Resolve coreference but annotate the actual named mention, not the pronoun.
+
+---
+
+## Event Classification
+
+### Event Classes
+
+| Class | Definition | Distinguishing criteria |
+|-------|-----------|------------------------|
+| `earnings_beat` | Reported EPS or revenue exceeds consensus | Explicit comparison to estimates |
+| `earnings_miss` | Reported EPS or revenue below consensus | Explicit comparison to estimates |
+| `guidance_raise` | Forward guidance raised vs prior or consensus | Future-looking, not historical result |
+| `guidance_cut` | Forward guidance lowered | Future-looking, not historical result |
+| `ma_announcement` | Merger, acquisition, investment, or divestiture | Transaction between entities |
+| `legal_regulatory` | Lawsuit, fine, regulatory action, or settlement | Legal or regulatory body involved |
+| `product_launch` | New product, service, or major feature announced | Not routine updates |
+| `supply_chain` | Disruption, partnership, or change in supply relationships | Affects production/delivery |
+| `rating_change` | Analyst upgrade, downgrade, or target change | From research analyst/firm |
+| `management_change` | CEO/CFO/board appointment, resignation, or removal | C-suite or board level |
+| `macro_event` | Interest rates, policy, trade, geopolitical | Not specific to one company |
+| `dividend_change` | Dividend increase, decrease, or special dividend | Shareholder distribution |
+| `buyback` | Share repurchase program announcement or completion | Capital return via buyback |
+
+### Adjudication triggers for events
+
+Route to the 9B adjudicator when:
+- The same facts could be classified as multiple event types (e.g., guidance_raise during an earnings call could be either earnings_beat or guidance_raise — label the most specific applicable class).
+- The event is implied but not explicitly stated.
+- The primary company is unclear.
+
+### Positive example
+
+```
+Source: "Apple beat earnings expectations with EPS of $1.52 vs $1.43 expected"
+Event class: earnings_beat
+Confidence: 0.98
+```
+
+### Negative example
+
+```
+Source: "Apple reported EPS of $1.52"
+Event class: earnings_beat
+```
+❌ Without a comparison to consensus/estimates, this is a numeric fact report, not an earnings beat. The document must provide evidence of beating expectations.
+
+---
+
+## Relations
+
+### Relation Types
+
+| Type | Subject | Object | When to use |
+|------|---------|--------|-------------|
+| `directly_affects` | Event | Company | Event explicitly names or discusses the company |
+| `inferred_exposure` | Event | Company | Exposure inferred from sector, supply chain, or competition |
+| `competes_with` | Company | Company | Competitive relationship stated or clearly implied |
+| `supplies` | Company | Company | Supply chain relationship stated |
+
+### Critical distinction: directly_affects vs inferred_exposure
+
+- `directly_affects`: The document **explicitly states** the company is impacted. Evidence span exists.
+- `inferred_exposure`: The impact is **reasoned** from relationships, not stated. May have weak or no direct evidence span.
+
+Only `directly_affects` enters primary company extraction. `inferred_exposure` flows through the separate interpolation/propagation architecture with distinct confidence and provenance.
+
+### Positive example (directly_affects)
+
+```
+Source: "Microsoft announced a $10 billion investment in OpenAI"
+Relation: directly_affects(event=ma_announcement, company=Microsoft)
+Evidence: "Microsoft announced"
+```
+
+### Negative example (incorrectly using directly_affects)
+
+```
+Source: "Microsoft announced a $10 billion investment in OpenAI"
+Relation: directly_affects(event=ma_announcement, company=Google)
+```
+❌ Google is not mentioned in the event sentence. This should be `inferred_exposure` based on competitive relationship, with appropriate lower confidence.
+
+---
+
+## Numeric Facts
+
+### Annotation rules
+
+1. Always store both `literal_value` (exact text) and `normalized_value` (parsed number).
+2. Include `unit` (USD, %, bps, shares, etc.).
+3. Link to the subject entity when determinable.
+4. Use `predicate` to capture the semantic role: reported, expected, raised_to, cut_to, beat_by, missed_by.
+5. Include `period` when the fact references a specific time frame.
+
+### Normalization conventions
+
+| Literal | Normalized | Unit |
+|---------|-----------|------|
+| "$1.52" | 1.52 | USD |
+| "$94.9 billion" | 94900000000 | USD |
+| "25 basis points" | 0.25 | percentage_points |
+| "4%" | 4.0 | % |
+| "$0.26 per share" | 0.26 | USD |
+
+### Positive example
+
+```json
+{
+  "fact_type": "eps",
+  "predicate": "reported",
+  "literal_value": "$1.52 per share",
+  "normalized_value": 1.52,
+  "unit": "USD",
+  "period": {"period_type": "fiscal_quarter", "fiscal_year": 2025, "fiscal_quarter": 1}
+}
+```
+
+### Negative example
+
+```json
+{
+  "fact_type": "eps",
+  "predicate": "reported",
+  "literal_value": "$1.52 per share",
+  "normalized_value": 152,
+  "unit": "cents"
+}
+```
+❌ While $1.52 = 152 cents, always normalize to the unit stated in the source. Conversion to a different unit introduces potential confusion.
+
+---
+
+## Sentiment
+
+### Rules
+
+1. Sentiment is **company-specific**, not document-level. A single article can have positive sentiment for one company and negative for another.
+2. Annotate probability distributions (positive, negative, neutral) that sum to 1.0.
+3. `mixed` label is used when evidence groups disagree — it is computed from evidence-group-level disagreement, NOT an unconstrained fourth class.
+4. The label should reflect the dominant probability.
+
+### When to label "mixed"
+
+Label `mixed` when:
+- Different paragraphs contain opposing sentiment for the same company
+- The same fact has both positive and negative implications (e.g., restructuring = cost cuts but also layoffs)
+- Analyst opinions explicitly disagree within the document
+
+Do NOT label `mixed` when:
+- Sentiment is merely uncertain or mild — that's `neutral` with lower confidence
+- The document discusses multiple companies with different sentiments — annotate separately per company
+
+### Positive example
+
+```json
+{
+  "label": "mixed",
+  "positive_probability": 0.40,
+  "negative_probability": 0.45,
+  "neutral_probability": 0.15,
+  "evidence_ids": ["ev-pressure", "ev-validation"]
+}
+```
+(Article says AI investment pressures cloud revenue but validates the broader thesis)
+
+### Negative example
+
+```json
+{
+  "label": "mixed",
+  "positive_probability": 0.85,
+  "negative_probability": 0.05,
+  "neutral_probability": 0.10
+}
+```
+❌ When positive_probability dominates at 0.85, the label should be `positive`, not `mixed`. Mixed requires genuine disagreement in evidence.
+
+---
+
+## Direct Effects vs Inferred Exposure
+
+### Direct Effects
+
+A direct effect means the document **explicitly states or clearly demonstrates** that an event impacts a specific company.
+
+**Criteria:**
+- The company is named in the same sentence or paragraph as the event
+- The causal link is stated, not inferred
+- Evidence span directly connects event to company
+
+### Inferred Exposure
+
+Inferred exposure captures **reasoned but unstated** impacts on companies.
+
+**Criteria:**
+- The company is NOT explicitly linked to the event in the source text
+- The connection comes from known relationships (competitor, supplier, sector peer)
+- Confidence should be lower than direct effects (typically 0.5–0.8)
+- Requires `reasoning` field explaining the inference chain
+
+### Adjudication routing
+
+When it's unclear whether an effect is direct or inferred, mark an ambiguity marker with type `implied_causal_impact` and route to the 9B adjudicator.
+
+---
+
+## Ambiguity Markers
+
+### When to flag
+
+Flag ambiguity when:
+- An alias resolves to multiple candidate companies (`unresolved_alias`)
+- Multiple companies could be the primary subject (`multiple_primary_companies`)
+- Numeric facts within the same document contradict each other (`contradictory_numeric_facts`)
+- Sentiment evidence points in opposing directions for the same company (`conflicting_sentiment`)
+- Impact is implied through causal chain, not stated (`implied_causal_impact`)
+- Guidance must be compared to consensus to determine direction (`guidance_vs_consensus_requires_reasoning`)
+- A required field cannot be determined from available evidence (`material_field_missing`)
+- Evidence covers less than the minimum threshold for confident extraction (`evidence_coverage_below_threshold`)
+- Calibrated confidence falls below the routing threshold (`calibrated_confidence_below_threshold`)
+- A relation spans multiple document chunks (`long_document_cross_chunk_relation`)
+
+### Severity levels
+
+- **low**: The annotation is likely correct but has reduced certainty. Fast path may proceed with a confidence penalty.
+- **medium**: The annotation requires review. Routes to adjudication by default.
+- **high**: The annotation cannot be reliably made without semantic reasoning. Always routes to adjudication.
+
+---
+
+## Safety-Critical Fields
+
+The following fields are **safety-critical** for promotion gates. Errors in these fields can directly cause incorrect trading decisions:
+
+| Field | Why it's critical | Minimum promotion gate |
+|-------|-------------------|----------------------|
+| Company identity (ticker) | Wrong ticker = trade on wrong security | Precision ≥ 0.95, Recall ≥ 0.90 |
+| Event class | Misclassifying beat/miss inverts signal direction | Macro-F1 ≥ 0.85 |
+| Sentiment direction | Wrong sentiment → wrong position direction | Direction accuracy ≥ 0.90 |
+| Numeric fact values | Wrong magnitude affects impact estimation | Tolerance match ≥ 0.92 |
+| Direct effect attribution | Wrong company attribution creates false signals | Precision ≥ 0.93 |
+| Evidence support | Unsupported claims are unverifiable | Support rate ≥ 0.95 |
+| Confidence calibration | Overconfidence bypasses review | ECE ≤ 0.05 |
+
+Annotators must pay special attention to these fields. During review, any error in a safety-critical field requires correction before the annotation can receive "gold" status.
+
+---
+
+## Annotation Workflow
+
+1. **First pass:** Identify all entities and evidence spans
+2. **Second pass:** Classify events and link to companies
+3. **Third pass:** Extract numeric facts with periods
+4. **Fourth pass:** Assess per-company sentiment
+5. **Fifth pass:** Identify relations, direct effects, and inferred exposures
+6. **Sixth pass:** Flag ambiguities and set confidence levels
+7. **Review:** Senior annotator validates safety-critical fields
+
+### Inter-annotator agreement
+
+Hard cases (flagged with ambiguity markers) receive double annotation. Inter-annotator agreement is measured per field type using Cohen's kappa. Target: κ ≥ 0.80 for entity and event labels, κ ≥ 0.70 for relations and sentiment.
+
+---
+
+## Version History
+
+| Version | Date | Changes |
+|---------|------|---------|
+| 1.0.0 | 2025-01-15 | Initial schema and guidelines |

docs/notes/session-context-2026-07-11.md

@@ -0,0 +1,84 @@
+# Session Context — July 11, 2026
+
+## Current State
+
+### Active Namespace: `stonks-beta`
+- This is the ONLY namespace that should be running
+- `stonks-oracle` namespace has been scaled to 0 replicas (all deployments)
+- Dashboard: `https://stonks-beta.celestium.life`
+- API: `https://stonks-api-beta.celestium.life`
+
+### What Was Done This Session
+
+#### 1. Pipeline Health Fixes (spec: `.kiro/specs/pipeline-health-fixes/`)
+All implemented and deployed:
+- **Stuck Parsed Docs**: `STALE_PARSED_THRESHOLD_MINUTES` 240→30, `LIMIT` 100→500, `_ENQUEUED_TTL` 14400→3600 (`services/scheduler/app.py`)
+- **Price Fallback**: Added 24h market_snapshots time-window fallback in `create_prediction_snapshot()` (`services/validation/prediction_snapshot.py`)
+- **Sentiment Normalization**: Added `normalize_impact_scores()` z-score function (`services/aggregation/scoring.py`) + integrated into `aggregate_company_window()` (`services/aggregation/worker.py`)
+- **Signal Engine**: Replicas set to 0 in all Helm values files
+- **Quality Gate**: `max_snapshot_age_hours` 24→48 (`services/trading/model_quality_gate.py`)
+- **Backfill script**: `scripts/backfill_snapshot_prices.py` (one-time, not yet run on beta)
+
+#### 2. Extractor Null-Field Fix
+- `services/extractor/schemas.py`: `_normalize_extraction_data()` now handles `None` values (not just missing keys) and filters out company entries with empty ticker
+- Test updated: `tests/test_extractor_schemas.py::test_validate_semantic_missing_ticker_is_error`
+
+#### 3. Macro Doc Status Fix
+- `services/extractor/main.py`: `_process_macro_classification()` now updates document status to 'extracted' on success, 'extraction_failed' on error
+- Beta DB: manually fixed 1849 stuck macro docs (UPDATE status='extracted' WHERE id IN global_events)
+
+#### 4. Dashboard Fix
+- `frontend/src/pages/OpsPipeline.tsx`: Document Stages now uses time-filtered `/health` data (consistent with other sections), all-time from SSE stream shown as subtitle, time range labels added to all sections
+
+#### 5. CI/CD DNS Fix
+- `.woodpecker/*.yml`: All 5 pipeline files now use `clone.git.settings.remote: http://10.43.73.77:3000/admin/stonks-oracle.git` (Gitea ClusterIP directly, bypasses DNS)
+- CoreDNS: scaled to 4 replicas, `forward . 192.168.42.1`, `dnsPolicy: None` with `nameservers: [192.168.42.1]`
+- Woodpecker: `WOODPECKER_BACKEND_K8S_DNS_CONFIG` has `nameservers:[10.43.0.10]` + searches including `git-server.svc.cluster.local`
+
+### Known Issues / TODO
+
+1. **`stonks-oracle` namespace**: Scaled to 0 but still exists with stale data (42K extraction queue in Redis DB 0). Could be cleaned up or deleted entirely.
+
+2. **Thesis Rewriter agent**: Was hammering vLLM from stonks-oracle namespace (5600+ calls/24h). Now stopped since namespace is scaled down. If it was also running in beta, check if recommendation service is calling vLLM for thesis rewrites excessively.
+
+3. **`AxionML/Qwen3.5-9B-NVFP4` requests**: Something external is hitting vLLM with a model that doesn't exist (404s). Not from our pipeline — likely Open WebUI or another tool on the network configured with wrong model name. Source IP: goes through `vllm-metrics` nginx proxy (`10.42.1.155`).
+
+4. **GitHub mirror**: `finalize.yml` mirror-github step fails (SSH key or DNS). Has `failure: ignore` so non-blocking. Needs `github_ssh_key` secret configured in Woodpecker.
+
+5. **OpsPipeline dashboard**: Numbers now show time-filtered data. The "Document Stages" section shows counts from the selected time window (default 24h), with all-time totals as subtle subtitles. Currently beta shows: extracted=5417, low_quality=1659, parsed=15.
+
+6. **Aggregation not generating trends on weekends**: Expected — market hours check prevents weekend trend generation. Will resume Monday.
+
+7. **15 docs still in `parsed` status**: These are likely fresh ingests waiting for the next extraction cycle. Not stuck.
+
+### Agent Performance (beta, last 24h as of session end)
+- Document Intelligence Extractor: 33 calls, 94% success, avg 11.4s, conf 0.794
+- Global Event Classifier: 81 calls, 99% success, avg 4.2s, conf 0.745
+- Thesis Rewriter: 5603 calls, 100% success, avg 2.5s (from stonks-oracle before shutdown)
+- Report Summarizer: 6 calls, 100% success, avg 6.9s
+
+### Infrastructure
+- k3s cluster: 4 NixOS nodes (gremlin-1 through gremlin-4)
+- vLLM: `vllm-service` namespace, model `numind/NuExtract3`, 4070 Ti Super 16GB
+- CoreDNS: 4 replicas, `forward . 192.168.42.1`
+- Redis: DB 0 = stonks-oracle (stale), DB 1 = stonks-beta (active)
+- PostgreSQL: shared instance, both namespaces use same DB server (different databases? or same? — needs verification)
+- Gitea: `git-server` namespace, ClusterIP 10.43.73.77:3000, NodePort 30300
+- Woodpecker: `woodpecker` namespace, kubernetes backend, 2 agents
+
+### Key Files Modified
+```
+services/scheduler/app.py          — recovery thresholds + batch limit
+services/validation/prediction_snapshot.py — 24h price fallback
+services/aggregation/scoring.py    — normalize_impact_scores()
+services/aggregation/worker.py     — normalization integration
+services/trading/model_quality_gate.py — 48h threshold
+services/extractor/schemas.py      — null field handling
+services/extractor/main.py         — macro doc status update
+frontend/src/pages/OpsPipeline.tsx  — dashboard fix
+scripts/backfill_snapshot_prices.py — new script
+tests/test_pbt_pipeline_health_*.py — PBT tests
+tests/test_extractor_schemas.py    — updated test
+infra/helm/stonks-oracle/values*.yaml — signal-engine replicas
+.woodpecker/*.yml                  — ClusterIP clone fix
+```

docs/overview-for-investors.md

@@ -0,0 +1,144 @@
+# Stonks Oracle — What It Is and What It Does
+
+## The One-Liner
+
+Stonks Oracle is an autonomous market intelligence system that reads the news so you don't have to, forms a view on 50 publicly traded companies, and paper-trades that view — then grades its own homework.
+
+---
+
+## The Problem It Solves
+
+Markets are noisy. Every day, hundreds of news articles, SEC filings, earnings transcripts, and geopolitical headlines hit the wire. A human analyst covering even a dozen names struggles to weigh all of it in real time. Most retail and even some institutional desks end up reacting to headlines rather than synthesizing the full picture.
+
+Stonks Oracle replaces that manual synthesis with an always-on pipeline:
+
+1. **It reads everything.** News articles, 10-K/10-Q filings, earnings calls, press releases, and macro/geopolitical headlines — ingested automatically on a schedule.
+2. **It extracts structured intelligence.** A local AI model reads each document and pulls out: which companies are mentioned, the sentiment (bullish / bearish / neutral), the catalyst type (earnings, product launch, regulatory action, M&A, etc.), impact horizon (same-day through 90 days), key facts, and material risks.
+3. **It forms a view.** Those individual extractions are aggregated into rolling trend summaries per company, refreshed continuously. The system flags contradictions (e.g., one filing is bullish but a news article is bearish) and tracks confidence based on evidence depth.
+4. **It decides whether to trade.** When confidence is high enough, contradiction is low, and evidence is fresh, it issues a buy or sell recommendation — with a full written thesis explaining why.
+5. **It executes paper trades.** An autonomous trading engine places orders through Alpaca's paper-trading system. Position sizing, stop-losses, take-profits, sector concentration limits, and circuit breakers are all built in.
+6. **It measures itself.** Every prediction is frozen at the moment it's made, then checked against actual price movements days and weeks later. The system tracks its own win rate, calibration, and whether it's beating SPY.
+
+---
+
+## The Universe
+
+50 companies across 10 sectors:
+
+| Sector | Examples |
+|--------|----------|
+| Technology | AAPL, MSFT, NVDA, GOOGL, META |
+| Consumer Cyclical | AMZN, TSLA, NKE, SBUX |
+| Financial Services | JPM, GS, V, MA |
+| Healthcare | JNJ, UNH, PFE, LLY |
+| Energy | XOM, CVX, COP |
+| Communication Services | NFLX, DIS, T |
+| Industrials | CAT, BA, UPS |
+| Consumer Defensive | PG, KO, WMT |
+| Real Estate | AMT, PLD |
+| Utilities | NEE, DUK |
+
+46 competitor relationships are defined (direct rivals, same-sector peers, overlapping products, supply chain adjacencies) so the system can propagate signals — e.g., if a semiconductor shortage hits one chipmaker, the system assesses exposure for its competitors and supply chain partners.
+
+---
+
+## The Three Signal Layers
+
+Think of these as three analysts sitting at the same desk, each watching a different feed:
+
+### Layer 1 — Company-Specific Intelligence
+
+The bread and butter. Every news article and filing about a specific company gets scored for sentiment, impact magnitude, and time horizon. These signals are weighted by recency (yesterday's earnings matter more than last month's), source credibility, and novelty (the fifth article repeating the same news adds less information than the first).
+
+Trend summaries roll up across five windows: intraday, 1 day, 7 days, 30 days, and 90 days — giving both a "what's happening right now" and a "what's the longer arc" view.
+
+### Layer 2 — Macro & Geopolitical
+
+Global events (trade wars, rate decisions, geopolitical crises, commodity shocks) are classified by impact type and severity. Each company has an exposure profile — geographic revenue mix, supply chain regions, commodity dependencies — that maps macro events down to company-level impact scores.
+
+A tariff announcement on Chinese imports doesn't affect all 50 companies equally. Apple with its Chinese manufacturing exposure gets a higher impact score than Procter & Gamble with largely domestic supply chains.
+
+### Layer 3 — Competitive & Historical Patterns
+
+The system mines its own history: when this type of catalyst (say, an earnings beat) happened to this company in the past, what happened to the stock? What happened to its competitors? If NVIDIA reports a blowout quarter, does AMD tend to sell off or rally in sympathy?
+
+This layer also tracks major corporate actions (M&A, restructurings, leadership changes) and propagates their implications across the competitive web.
+
+**Safety rule:** The system never trades on macro or competitive signals alone. If there's no company-specific evidence supporting the thesis, the recommendation is downgraded to informational only.
+
+---
+
+## How a Trade Happens
+
+Here's the chain from "news article published" to "paper order placed":
+
+1. **Ingestion** — The article is fetched, deduplicated, and stored.
+2. **Parsing** — Raw HTML is cleaned, boilerplate is stripped, quality is scored.
+3. **Extraction** — The AI model reads the cleaned text and produces structured JSON: tickers mentioned, sentiment, catalysts, key facts, risks.
+4. **Aggregation** — The new extraction is merged into rolling trend summaries for each mentioned company. Confidence, contradiction, and evidence depth are recalculated.
+5. **Recommendation** — If the trend passes quality filters (enough evidence, high enough confidence, low enough contradiction, not stale), a BUY or SELL recommendation is generated with a written thesis.
+6. **Risk checks** — The trading engine asks: Is the circuit breaker tripped? Is the market open? Do I already have too many positions? Is this sector already overweight? Are earnings in the next 48 hours?
+7. **Position sizing** — Dollar amount is computed from confidence, portfolio heat, and the current risk tier (conservative / moderate / aggressive — auto-adjusted based on trailing performance).
+8. **Execution** — The order goes to Alpaca's paper-trading API. Stop-loss and take-profit levels are set automatically based on the stock's recent volatility.
+9. **Monitoring** — Open positions are tracked with trailing stops. If a position declines past its stop, it's closed. If it hits the take-profit target, it's closed.
+10. **Scoring** — Days later, the prediction is evaluated against the actual price move. Did the call go the right way? Did the confidence track reality?
+
+---
+
+## Risk Management (Built In, Not Bolted On)
+
+- **Circuit breakers** — If daily losses exceed a threshold or a single position loses too much, all trading halts automatically.
+- **Position caps** — No single position can consume more than a set percentage of the portfolio.
+- **Sector concentration limits** — The system won't pile into one sector even if all signals are bullish.
+- **Correlation awareness** — New positions are rejected if they'd push portfolio correlation too high.
+- **Earnings blackout** — Position sizes are reduced or skipped entirely within 48 hours of an earnings announcement.
+- **Reserve pool** — Profits are partially siphoned into an emergency liquidity reserve.
+- **Risk tier auto-adjustment** — The system evaluates its own Sharpe ratio, drawdown, and win rate daily and shifts between conservative, moderate, and aggressive modes.
+
+---
+
+## Self-Grading: The Validation Loop
+
+Most trading systems tell you their view. Few systematically check whether that view was right.
+
+Stonks Oracle captures every prediction as an immutable snapshot — the thesis, the confidence, the price at the time, the evidence cited. Then it waits. After the prediction's time horizon elapses (1 day, 7 days, 30 days), it compares the predicted direction against the actual price movement and computes:
+
+- **Win rate** — What fraction of directional calls were correct?
+- **Calibration** — When the system says "70% confident bullish," does the stock actually go up ~70% of the time? (If it only goes up 50% of the time, the system is overconfident.)
+- **Information coefficient** — Does the system's score have any linear correlation with actual returns?
+- **Excess return vs. SPY** — Is it adding alpha, or would you be better off in an index fund?
+- **Source attribution** — Which news sources and signal types actually contribute to correct predictions? Which are noise?
+
+If model quality drops below defined thresholds, a safety gate prevents the system from upgrading recommendations from "informational" to "paper eligible" — it forces itself to the sidelines until accuracy recovers.
+
+---
+
+## The Dashboard
+
+A web-based interface lets you see everything the system sees:
+
+- **Home** — Portfolio value, daily P&L, risk tier, active alerts.
+- **Companies** — The tracked universe with current trend summaries and signal strength.
+- **Documents** — Every ingested article and filing, with the AI's structured extraction visible.
+- **Trends** — Per-company trend charts across all time windows, with evidence chains you can click through.
+- **Recommendations** — Active and historical recommendations with full theses and risk classifications.
+- **Trading** — The engine's status: open positions, reserve pool, circuit breaker state, portfolio heat map.
+- **Orders & Positions** — Full trade blotter with execution details.
+- **Macro Events** — Global event timeline showing what the system is tracking at the geopolitical level.
+- **Reports** — AI-generated daily and weekly performance summaries.
+- **Model Performance** — Calibration curves, win rate trends, source reliability scores.
+- **SQL Explorer** — Ad-hoc queries against the full analytical data warehouse, with a chart builder.
+
+---
+
+## What It Is Not
+
+- **Not a live trading system (yet).** All trades are paper trades through Alpaca's sandbox. The architecture supports live execution, but safety gates and validation must demonstrate consistent edge before real money is at risk.
+- **Not a black box.** Every recommendation includes a full thesis, every trade has a decision trace, every prediction links back to the specific evidence that drove it.
+- **Not a prediction guarantee.** Markets are hard. The system's value is in disciplined synthesis, consistent process, and honest self-measurement — not in claiming to always be right.
+
+---
+
+## Where It's Headed
+
+Active development is upgrading the signal math from rule-based heuristics to probabilistic Bayesian inference — running both approaches in parallel, comparing their verdicts, and using the disagreements as training signals for continuous improvement. The goal is a system that not only reads the market but learns from its own track record which types of evidence, in which market regimes, actually predict future price moves.

frontend/.dockerignore

@@ -1,3 +1,2 @@
-node_modules
 dist
 .env

frontend/Dockerfile

@@ -1,9 +1,6 @@
-# Stage 1: Build
-# Base images: uses Harbor proxy cache in CI, falls back to Docker Hub externally
+# Stage 1: Build (node_modules pre-installed by CI step outside sandbox)
 FROM registry.celestium.life/dockerhub-cache/library/node:24-alpine AS build
 WORKDIR /app
-COPY package.json package-lock.json ./
-RUN npm ci
 COPY . .
 ARG VITE_QUERY_API_URL=""
 ARG VITE_SYMBOL_REGISTRY_URL=""

frontend/src/pages/Dashboards.tsx

@@ -125,7 +125,7 @@ function SymbolOverview({ ticker }: { ticker: string }) {
 // ---------------------------------------------------------------------------
 
 function SentimentHeatmap() {
-  const { data: trends, isLoading } = useTrends({ limit: 500 });
+  const { data: trends, isLoading } = useTrends({ limit: 200 });
   const { data: companies } = useCompanies();
 
   if (isLoading) return <LoadingSpinner />;
@@ -205,7 +205,7 @@ function SentimentHeatmap() {
 // ---------------------------------------------------------------------------
 
 function PredictionAccuracy() {
-  const { data: recs, isLoading } = useRecommendations({ limit: 500 });
+  const { data: recs, isLoading } = useRecommendations({ limit: 200 });
 
   if (isLoading) return <LoadingSpinner />;
   if (!recs?.length) return <p className="text-sm text-gray-500">No recommendation data available</p>;

frontend/src/pages/OpsPipeline.tsx

@@ -63,15 +63,18 @@ export function OpsPipelinePage() {
   const aggregation = (data?.aggregation ?? {}) as Record<string, unknown>;
   const pipelineEnabled = (data?.pipeline_enabled ?? true) as boolean;
 
-  // Prefer live stream data for queue depths and doc stages, fall back to initial fetch
+  // Queue depths come from live stream (always current)
+  // Document stages: show time-filtered from /health endpoint so numbers are consistent
+  // with Parsing/Extraction/Trends sections. Stream provides live queue data only.
   const queueDepths = stream?.queue_depths
     ?? (data?.queue_depths as Record<string, number> | undefined)
     ?? {};
-  const docStages = stream?.document_stages
-    ?? Object.fromEntries(
-      ((data?.document_stages as Array<{ status: string; doc_count: number }>) ?? [])
-        .map((s) => [s.status, s.doc_count]),
-    );
+  const docStages = Object.fromEntries(
+    ((data?.document_stages as Array<{ status: string; doc_count: number }>) ?? [])
+      .map((s) => [s.status, s.doc_count]),
+  );
+  // All-time totals from stream for reference
+  const allTimeStages = stream?.document_stages ?? {};
 
   const failedCount = docStages['extraction_failed'] ?? 0;
 
@@ -150,9 +153,13 @@ export function OpsPipelinePage() {
 
       {/* Document Stage Counts */}
       <Card>
-        <h2 className="mb-3 text-sm font-medium text-gray-400">Document Stages</h2>
+        <h2 className="mb-3 text-sm font-medium text-gray-400">
+          Document Stages
+          <span className="ml-2 text-xs text-gray-600">(last {hours}h)</span>
+        </h2>
         <div className="grid grid-cols-2 gap-3 sm:grid-cols-4">
           {Object.entries(docStages).map(([status, count]) => {
+            const allTime = allTimeStages[status];
             const color = status === 'extracted' ? 'text-green-400'
               : status === 'parsed' ? 'text-yellow-400'
               : status === 'extraction_failed' ? 'text-red-400'
@@ -162,6 +169,9 @@ export function OpsPipelinePage() {
               <div key={status} className="rounded-lg border border-surface-700 bg-surface-950 p-3 text-center">
                 <div className={`text-2xl font-bold tabular-nums ${color}`}>{count}</div>
                 <div className="text-xs capitalize text-gray-500">{status.replace('_', ' ')}</div>
+                {allTime != null && allTime !== count && (
+                  <div className="mt-0.5 text-[10px] text-gray-600">{allTime} total</div>
+                )}
               </div>
             );
           })}
@@ -170,7 +180,10 @@ export function OpsPipelinePage() {
 
       {/* Parsing Quality */}
       <Card>
-        <h2 className="mb-3 text-sm font-medium text-gray-400">Parsing Quality</h2>
+        <h2 className="mb-3 text-sm font-medium text-gray-400">
+          Parsing Quality
+          <span className="ml-2 text-xs text-gray-600">(last {hours}h)</span>
+        </h2>
         <dl className="grid grid-cols-2 gap-3 text-sm sm:grid-cols-5">
           <Stat label="Total Parsed" value={parsing.total_parsed} />
           <Stat label="High Confidence" value={parsing.high_confidence} color="text-green-400" />
@@ -182,7 +195,10 @@ export function OpsPipelinePage() {
 
       {/* Extraction Stats */}
       <Card>
-        <h2 className="mb-3 text-sm font-medium text-gray-400">Extraction Validation</h2>
+        <h2 className="mb-3 text-sm font-medium text-gray-400">
+          Extraction Validation
+          <span className="ml-2 text-xs text-gray-600">(last {hours}h)</span>
+        </h2>
         <dl className="grid grid-cols-2 gap-3 text-sm sm:grid-cols-5">
           <Stat label="Total" value={extraction.total_extractions} />
           <Stat label="Valid" value={extraction.valid} color="text-green-400" />
@@ -194,7 +210,10 @@ export function OpsPipelinePage() {
 
       {/* Aggregation */}
       <Card>
-        <h2 className="mb-3 text-sm font-medium text-gray-400">Trend Generation</h2>
+        <h2 className="mb-3 text-sm font-medium text-gray-400">
+          Trend Generation
+          <span className="ml-2 text-xs text-gray-600">(last {hours}h)</span>
+        </h2>
         <dl className="grid grid-cols-2 gap-3 text-sm sm:grid-cols-4">
           <Stat label="Trends Generated" value={aggregation.trends_generated} />
           <Stat label="Symbols Covered" value={aggregation.symbols_covered} />

infra/helm/stonks-oracle/templates/specialist-deployment.yaml

@@ -0,0 +1,108 @@
+{{- if .Values.specialist }}
+{{- if .Values.specialist.enabled }}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: specialist
+  namespace: {{ .Release.Namespace }}
+  labels:
+    app: specialist
+    {{- include "stonks.labels" . | nindent 4 }}
+    stonks-oracle/tier: processing
+spec:
+  replicas: {{ .Values.specialist.replicas | default 2 }}
+  selector:
+    matchLabels:
+      app: specialist
+  template:
+    metadata:
+      labels:
+        app: specialist
+        stonks-oracle/tier: processing
+    spec:
+      automountServiceAccountToken: false
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      securityContext:
+        {{- include "stonks.podSecurityContext" . | nindent 8 }}
+      containers:
+        - name: specialist
+          image: {{ .Values.image.registry }}/specialist:{{ .Values.image.tag }}
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          command: ["sh", "-c", "uvicorn services.specialist.app:app --host 0.0.0.0 --port 8000"]
+          ports:
+            - containerPort: 8000
+          env:
+            - name: SPECIALIST_MODEL
+              value: {{ .Values.specialist.model | default "urchade/gliner_large-v2.1" | quote }}
+            - name: SPECIALIST_MAX_BATCH_SIZE
+              value: {{ .Values.specialist.maxBatchSize | default "32" | quote }}
+            - name: SPECIALIST_MAX_WAIT_MS
+              value: {{ .Values.specialist.maxWaitMs | default "50.0" | quote }}
+            - name: SPECIALIST_MAX_QUEUE_SIZE
+              value: {{ .Values.specialist.maxQueueSize | default "256" | quote }}
+            - name: SPECIALIST_TEST_MODE
+              value: {{ .Values.specialist.testMode | default "0" | quote }}
+          securityContext:
+            {{- include "stonks.containerSecurityContext" . | nindent 12 }}
+          envFrom:
+            - configMapRef:
+                name: stonks-config
+            {{- range .Values.specialist.secrets }}
+            - secretRef:
+                name: {{ . }}
+            {{- end }}
+          resources:
+            requests:
+              cpu: {{ .Values.specialist.resources.requests.cpu | default "2" | quote }}
+              memory: {{ .Values.specialist.resources.requests.memory | default "4Gi" }}
+            limits:
+              cpu: {{ .Values.specialist.resources.limits.cpu | default "6" | quote }}
+              memory: {{ .Values.specialist.resources.limits.memory | default "10Gi" }}
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: 8000
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: 8000
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 5
+          volumeMounts:
+            - name: tmp
+              mountPath: /tmp
+            - name: model-cache
+              mountPath: /root/.cache
+      volumes:
+        - name: tmp
+          emptyDir:
+            sizeLimit: 10Mi
+        - name: model-cache
+          emptyDir:
+            sizeLimit: 5Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: specialist
+  namespace: {{ .Release.Namespace }}
+  labels:
+    app: specialist
+    {{- include "stonks.labels" . | nindent 4 }}
+spec:
+  selector:
+    app: specialist
+  ports:
+    - port: 8000
+      targetPort: 8000
+      protocol: TCP
+{{- end }}
+{{- end }}

infra/helm/stonks-oracle/values-beta.yaml

@@ -17,6 +17,8 @@ services:
     replicas: 1
   dashboard:
     replicas: 1
+  signalEngine:
+    replicas: 0
 
 ## Beta-specific config overrides
 ## Beta shares the paper DB — DEPLOY_STAGE=beta isolates Redis keys
@@ -39,10 +41,10 @@ config:
   MINIO_SECURE: "false"
   BROKER_MODE: "paper"
   BROKER_PROVIDER: "alpaca"
-  OLLAMA_BASE_URL: "http://192.168.42.254:11434"
-  OLLAMA_MODEL: "qwen3.6"
-  VLLM_BASE_URL: "http://192.168.42.254:8000"
-  VLLM_MODEL: "AxionML/Qwen3.5-9B-NVFP4"
+  OLLAMA_BASE_URL: "http://nuextract-external.vllm-service.svc.cluster.local:2701"
+  OLLAMA_MODEL: "numind/NuExtract3"
+  VLLM_BASE_URL: "http://nuextract-external.vllm-service.svc.cluster.local:2701"
+  VLLM_MODEL: "numind/NuExtract3"
   VLLM_TIMEOUT: "120"
   VLLM_MAX_RETRIES: "2"
   VLLM_TEMPERATURE: "0.7"
@@ -58,10 +60,10 @@ secrets:
     POSTGRES_PASSWORD: "St0nks0racl3!"
     MINIO_ACCESS_KEY: "AKIA6V7J3N9B5P0D2YQH"
     MINIO_SECRET_KEY: "8fG3!v2rJ7$wN@9mLpQ6zXbC4tKdPqW1"
-    REDIS_PASSWORD: "PSCh4ng3me!"
+    REDIS_PASSWORD: "SGaIkaf342Ljym7jChmqPY7i"
   broker:
-    BROKER_API_KEY: "PKRTP2PRRNCO3AYRGCK2FGWGMJ"
-    BROKER_API_SECRET: "dWhCubuyzTGDTPqtV1HXdUGhu8ZQB6EP4oui3GRyDTT"
+    BROKER_API_KEY: "PK2SC5XYTYOXZZ7JZDZZPG3JMD"
+    BROKER_API_SECRET: "3kpUW5Wz7bjLVnbdygeN24NNoHtvK4d1bQKwNg53SPZw"
     BROKER_BASE_URL: "https://paper-api.alpaca.markets"
   market:
     MARKET_DATA_API_KEY: "NPwKtrLvoBxcKt3Byp5PEvuZiBZU_d8E"

infra/helm/stonks-oracle/values-live-math.yaml

@@ -26,7 +26,7 @@ secrets:
     POSTGRES_PASSWORD: "St0nks0racl3!"
     MINIO_ACCESS_KEY: "AKIA6V7J3N9B5P0D2YQH"
     MINIO_SECRET_KEY: "8fG3!v2rJ7$wN@9mLpQ6zXbC4tKdPqW1"
-    REDIS_PASSWORD: "PSCh4ng3me!"
+    REDIS_PASSWORD: "SGaIkaf342Ljym7jChmqPY7i"
   broker:
     BROKER_API_KEY: "PK64RS7NH24XPBI3IDEU3BB72Y"
     BROKER_API_SECRET: "Ho4D84392vB4s2TkGi52ra5FcxEskGfJSZYRKHa3qrYq"

infra/helm/stonks-oracle/values-paper.yaml

@@ -16,7 +16,10 @@ config:
   REDIS_DB: "2"
   DEPLOY_STAGE: "paper"
   POSTGRES_USER: "stonks_paper"
-  OLLAMA_BASE_URL: "http://10.1.1.12:2701"
+  OLLAMA_BASE_URL: "http://nuextract-external.vllm-service.svc.cluster.local:2701"
+  OLLAMA_MODEL: "numind/NuExtract3"
+  VLLM_BASE_URL: "http://nuextract-external.vllm-service.svc.cluster.local:2701"
+  VLLM_MODEL: "numind/NuExtract3"
   MARKET_DATA_BASE_URL: "https://api.polygon.io"
 
 ## Secrets — all credentials for the paper stage
@@ -27,7 +30,7 @@ secrets:
     POSTGRES_PASSWORD: "St0nks0racl3!"
     MINIO_ACCESS_KEY: "AKIA6V7J3N9B5P0D2YQH"
     MINIO_SECRET_KEY: "8fG3!v2rJ7$wN@9mLpQ6zXbC4tKdPqW1"
-    REDIS_PASSWORD: "PSCh4ng3me!"
+    REDIS_PASSWORD: "SGaIkaf342Ljym7jChmqPY7i"
   broker:
     BROKER_API_KEY: "PKECQBNHD6ZLKEXZZVIFTOLX72"
     BROKER_API_SECRET: "5pV8zfUn92zAUL4TAwFor3Lk8RqNBcRzN12Y1HJjU7Gn"
@@ -49,3 +52,5 @@ ingress:
 services:
   extractor:
     replicas: 1
+  signalEngine:
+    replicas: 0

infra/helm/stonks-oracle/values.yaml

@@ -19,8 +19,8 @@ services:
     tier: orchestration
     secrets: [stonks-core-secrets]
     resources:
-      requests: { cpu: 50m, memory: 64Mi }
-      limits: { cpu: 200m, memory: 128Mi }
+      requests: { cpu: 50m, memory: 128Mi }
+      limits: { cpu: 200m, memory: 256Mi }
 
   symbolRegistry:
     replicas: 1
@@ -99,8 +99,8 @@ services:
     port: 8000
     secrets: [stonks-core-secrets, stonks-broker-secrets, stonks-gmail-secrets]
     resources:
-      requests: { cpu: 100m, memory: 256Mi }
-      limits: { cpu: 500m, memory: 512Mi }
+      requests: { cpu: 100m, memory: 512Mi }
+      limits: { cpu: 1000m, memory: 1024Mi }
     probes:
       readiness: { path: /ready, port: 8000, initialDelay: 5, period: 10 }
       liveness: { path: /health, port: 8000, initialDelay: 10, period: 30 }
@@ -128,7 +128,7 @@ services:
       limits: { cpu: 200m, memory: 128Mi }
 
   signalEngine:
-    replicas: 1
+    replicas: 0
     pipeline: true
     image: signal-engine
     command: "python -m services.signal_engine.main"
@@ -139,7 +139,7 @@ services:
       limits: { cpu: 500m, memory: 256Mi }
 
   lakePublisher:
-    replicas: 1
+    replicas: 0
     pipeline: true
     image: lake-publisher
     command: "python -m services.lake_publisher.jobs"
@@ -185,15 +185,15 @@ config:
   REDIS_DB: "0"
   MINIO_ENDPOINT: "minio.minio-service.svc.cluster.local:80"
   MINIO_SECURE: "false"
-  OLLAMA_BASE_URL: "http://10.1.1.12:2701"
-  OLLAMA_MODEL: "qwen3.5:9b-fast"
+  OLLAMA_BASE_URL: "http://nuextract-external.vllm-service.svc.cluster.local:2701"
+  OLLAMA_MODEL: "numind/NuExtract3"
   OLLAMA_TIMEOUT: "240"
   OLLAMA_MAX_RETRIES: "2"
   OLLAMA_RETRY_BASE_DELAY: "1.0"
   OLLAMA_RETRY_MAX_DELAY: "10.0"
   OLLAMA_RETRY_BACKOFF_MULTIPLIER: "2.0"
-  VLLM_BASE_URL: "http://10.1.1.12:2701"
-  VLLM_MODEL: "qwen3.5:9b-fast"
+  VLLM_BASE_URL: "http://nuextract-external.vllm-service.svc.cluster.local:2701"
+  VLLM_MODEL: "numind/NuExtract3"
   VLLM_TIMEOUT: "120"
   VLLM_MAX_RETRIES: "2"
   VLLM_TEMPERATURE: "0.7"
@@ -240,7 +240,7 @@ secrets:
     POSTGRES_PASSWORD: "St0nks0racl3!"
     MINIO_ACCESS_KEY: "AKIA6V7J3N9B5P0D2YQH"
     MINIO_SECRET_KEY: "8fG3!v2rJ7$wN@9mLpQ6zXbC4tKdPqW1"
-    REDIS_PASSWORD: "PSCh4ng3me!"
+    REDIS_PASSWORD: "SGaIkaf342Ljym7jChmqPY7i"
   broker:
     BROKER_API_KEY: "PKECQBNHD6ZLKEXZZVIFTOLX72"
     BROKER_API_SECRET: "5pV8zfUn92zAUL4TAwFor3Lk8RqNBcRzN12Y1HJjU7Gn"
@@ -289,6 +289,24 @@ superset:
     requests: { cpu: 200m, memory: 512Mi }
     limits: { cpu: "1", memory: 2Gi }
 
+## Specialist inference service (CPU-first NER/classification)
+specialist:
+  enabled: true
+  replicas: 2
+  model: "urchade/gliner_large-v2.1"
+  maxBatchSize: "32"
+  maxWaitMs: "50.0"
+  maxQueueSize: "256"
+  testMode: "0"
+  secrets: [stonks-core-secrets]
+  resources:
+    requests:
+      cpu: "2"
+      memory: 4Gi
+    limits:
+      cpu: "6"
+      memory: 10Gi
+
 ## Network policies
 networkPolicies:
   enabled: true

infra/kube-vllm/deployment.yaml

@@ -0,0 +1,78 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: vllm
+  namespace: vllm-service
+  labels:
+    app: vllm
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: vllm
+  template:
+    metadata:
+      labels:
+        app: vllm
+    spec:
+      runtimeClassName: nvidia
+      enableServiceLinks: false
+      nodeSelector:
+        kubernetes.io/hostname: gremlin-1
+      containers:
+        - name: vllm
+          image: registry.celestium.life/stonks-oracle/vllm-openai:latest
+          imagePullPolicy: Always
+          args:
+            - "--model"
+            - "numind/NuExtract3"
+            - "--served-model-name"
+            - "numind/NuExtract3"
+            - "--host"
+            - "0.0.0.0"
+            - "--port"
+            - "8000"
+            - "--gpu-memory-utilization"
+            - "0.80"
+            - "--max-model-len"
+            - "8192"
+            - "--max-num-seqs"
+            - "8"
+          env:
+            - name: HF_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: vllm-secrets
+                  key: HF_TOKEN
+            - name: VLLM_ATTENTION_BACKEND
+              value: "FLASHINFER"
+          ports:
+            - containerPort: 8000
+              name: http
+          resources:
+            limits:
+              nvidia.com/gpu: "1"
+            requests:
+              cpu: "2"
+              memory: "8Gi"
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: 8000
+            initialDelaySeconds: 120
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: 8000
+            initialDelaySeconds: 300
+            periodSeconds: 30
+          volumeMounts:
+            - name: hf-cache
+              mountPath: /root/.cache/huggingface
+      volumes:
+        - name: hf-cache
+          persistentVolumeClaim:
+            claimName: vllm-hf-cache-pvc

infra/kube-vllm/monitor.sh

@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+VLLM="http://10.1.1.12:31508"
+GPU_HOST="root@10.1.1.12"
+INTERVAL=3
+
+while true; do
+  clear
+  echo "═══════════════════════════════════════════════════════════════"
+  echo "  vLLM MONITOR (K8s)  $(date '+%Y-%m-%d %H:%M:%S')"
+  echo "═══════════════════════════════════════════════════════════════"
+
+  # GPU
+  gpu=$(ssh -o ConnectTimeout=2 -o BatchMode=yes "$GPU_HOST" \
+    'nvidia-smi --query-gpu=name,temperature.gpu,power.draw,power.limit,memory.used,memory.total,utilization.gpu --format=csv,noheader,nounits' 2>/dev/null)
+  if [ -n "$gpu" ]; then
+    IFS=',' read -r name temp power power_cap mem_used mem_total gpu_util <<< "$gpu"
+    mem_free=$(awk "BEGIN{printf \"%.0f\", $mem_total-$mem_used}")
+    echo ""
+    echo "  GPU: ${name}"
+    echo "  ├─ Temp:  ${temp}°C    Power: ${power}W / ${power_cap}W"
+    echo "  ├─ VRAM:  ${mem_used} / ${mem_total} MiB  (${mem_free} MiB free)"
+    echo "  └─ Util:  ${gpu_util}%"
+  fi
+
+  # vLLM model info
+  models_json=$(curl -sf --max-time 2 "$VLLM/v1/models" 2>/dev/null)
+  if [ -n "$models_json" ]; then
+    echo ""
+    python3 -c "
+import json,sys
+data = json.loads(sys.argv[1])
+for m in data.get('data',[]):
+    print(f'  MODEL: {m[\"id\"]}')
+" "$models_json" 2>/dev/null
+  fi
+
+  # Prometheus metrics from vLLM /metrics endpoint
+  prom=$(curl -sf --max-time 2 "$VLLM/metrics" 2>/dev/null)
+  if [ -n "$prom" ]; then
+    python3 -c "
+import sys
+lines = sys.argv[1].split('\n')
+def gauge(prefix):
+    for l in lines:
+        if l.startswith(prefix) and not l.startswith('#'):
+            return float(l.split()[-1])
+    return 0
+def counter(prefix):
+    return sum(float(l.split()[-1]) for l in lines if l.startswith(prefix) and not l.startswith('#'))
+def histo_avg(prefix):
+    s = counter(prefix + '_sum')
+    c = counter(prefix + '_count')
+    return s/c if c > 0 else 0
+
+running = gauge('vllm:num_requests_running')
+waiting = gauge('vllm:num_requests_waiting')
+kv_pct  = gauge('vllm:gpu_cache_usage_perc') * 100
+prompt_tok = counter('vllm:prompt_tokens_total')
+gen_tok    = counter('vllm:generation_tokens_total')
+req_ok     = counter('vllm:request_success_total')
+preempts   = counter('vllm:num_preemptions_total')
+
+ttft   = histo_avg('vllm:time_to_first_token_seconds')
+itl    = histo_avg('vllm:inter_token_latency_seconds')
+e2e    = histo_avg('vllm:e2e_request_latency_seconds')
+tok_s  = 1/itl if itl > 0 else 0
+
+print()
+print('  REQUESTS:')
+print(f'  ├─ Running:    {int(running)}    Waiting: {int(waiting)}')
+print(f'  ├─ Completed:  {int(req_ok)}     Preemptions: {int(preempts)}')
+print(f'  └─ KV Cache:   {kv_pct:.1f}%')
+print()
+print('  TOKENS:')
+print(f'  ├─ Prompt:     {int(prompt_tok):,}')
+print(f'  └─ Generated:  {int(gen_tok):,}')
+print()
+print('  LATENCY:')
+print(f'  ├─ TTFT:       {ttft*1000:.0f}ms')
+print(f'  ├─ ITL:        {itl*1000:.1f}ms')
+print(f'  ├─ Tok/s:      {tok_s:.1f}')
+print(f'  └─ E2E avg:    {e2e:.2f}s')
+" "$prom" 2>/dev/null
+  else
+    echo ""
+    echo "  METRICS: unreachable"
+  fi
+
+  echo ""
+  echo "═══════════════════════════════════════════════════════════════"
+  echo "  Ctrl+C to exit"
+  sleep "$INTERVAL"
+done

infra/kube-vllm/service.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: nuextract
+  namespace: vllm-service
+spec:
+  ports:
+    - name: http
+      port: 8000
+      targetPort: 8000
+  selector:
+    app: vllm
+  type: ClusterIP
+---
+# External access via NodePort (like ollama's port 2701 pattern)
+apiVersion: v1
+kind: Service
+metadata:
+  name: nuextract-external
+  namespace: vllm-service
+spec:
+  ports:
+    - nodePort: 31508
+      name: vllm-api
+      port: 2701
+      targetPort: 8080
+  selector:
+    app: vllm-metrics
+  type: LoadBalancer

infra/kube-vllm/storage.yaml

@@ -0,0 +1,35 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: vllm-hf-cache-pv
+spec:
+  capacity:
+    storage: 50Gi
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: local-path
+  hostPath:
+    path: /var/lib/vllm/hf-cache
+    type: DirectoryOrCreate
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - gremlin-1
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: vllm-hf-cache-pvc
+  namespace: vllm-service
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 50Gi
+  storageClassName: local-path

infra/kube-vllm/values.yaml

@@ -0,0 +1,45 @@
+replicaCount: 1
+
+image:
+  repository: vllm/vllm-openai
+  tag: latest
+  pullPolicy: Always
+
+resources:
+  limits:
+    nvidia.com/gpu: 1
+  requests:
+    cpu: "2"
+    memory: "8Gi"
+
+runtimeClassName: nvidia
+
+env:
+  - name: HF_TOKEN
+    valueFrom:
+      secretKeyRef:
+        name: vllm-secrets
+        key: HF_TOKEN
+
+args:
+  - "serve"
+  - "numind/NuExtract3"
+  - "--served-model-name"
+  - "numind/NuExtract3"
+  - "--host"
+  - "0.0.0.0"
+  - "--port"
+  - "8000"
+  - "--gpu-memory-utilization"
+  - "0.45"
+  - "--max-model-len"
+  - "8192"
+  - "--max-num-seqs"
+  - "8"
+
+service:
+  type: ClusterIP
+  port: 8000
+
+nodeSelector:
+  kubernetes.io/hostname: gremlin-1

infra/kube-vllm/vllm-metrics.yaml

@@ -0,0 +1,74 @@
+# vLLM metrics proxy — similar to ollama-metrics
+# Proxies requests and exposes Prometheus metrics
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: vllm-metrics
+  namespace: vllm-service
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: vllm-metrics
+  template:
+    metadata:
+      labels:
+        app: vllm-metrics
+    spec:
+      containers:
+        - name: proxy
+          image: nginx:alpine
+          ports:
+            - containerPort: 8080
+          volumeMounts:
+            - name: nginx-conf
+              mountPath: /etc/nginx/conf.d/default.conf
+              subPath: default.conf
+      volumes:
+        - name: nginx-conf
+          configMap:
+            name: vllm-proxy-config
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: vllm-proxy-config
+  namespace: vllm-service
+data:
+  default.conf: |
+    upstream vllm_backend {
+        server nuextract.vllm-service.svc.cluster.local:8000;
+    }
+
+    server {
+        listen 8080;
+
+        # API proxy
+        location / {
+            proxy_pass http://vllm_backend;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_connect_timeout 300s;
+            proxy_read_timeout 300s;
+            proxy_send_timeout 300s;
+        }
+
+        # vLLM exposes /metrics natively
+        location /metrics {
+            proxy_pass http://vllm_backend/metrics;
+        }
+    }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: vllm-metrics
+  namespace: vllm-service
+spec:
+  ports:
+    - name: proxy
+      port: 8080
+      targetPort: 8080
+  selector:
+    app: vllm-metrics
+  type: ClusterIP

infra/migrations/040_inference_registry.sql

@@ -0,0 +1,122 @@
+-- Migration 040: Inference Registry
+-- Creates tables for the capability-aware inference gateway:
+--   inference_endpoints, model_deployments, agent_stage_bindings
+-- Adds lineage columns to agent_performance_log for v3 provenance tracking.
+
+-- ─── Helper: auto-update updated_at on row modification ───────────────────────
+CREATE OR REPLACE FUNCTION update_updated_at_column()
+RETURNS TRIGGER AS $$
+BEGIN
+    NEW.updated_at = now();
+    RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+-- ─── inference_endpoints ──────────────────────────────────────────────────────
+-- Stores registered inference service endpoints (Ollama, vLLM, OpenAI-compat, specialist).
+CREATE TABLE IF NOT EXISTS inference_endpoints (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    name TEXT NOT NULL UNIQUE,
+    protocol TEXT NOT NULL CHECK (protocol IN ('ollama_native', 'openai_chat', 'specialist_http')),
+    base_url TEXT NOT NULL,
+    auth_secret_ref TEXT,
+    auth_scheme TEXT NOT NULL DEFAULT 'bearer',
+    default_headers JSONB NOT NULL DEFAULT '{}',
+    health_path TEXT,
+    enabled BOOLEAN NOT NULL DEFAULT TRUE,
+    revision INTEGER NOT NULL DEFAULT 1,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_inference_endpoints_protocol
+    ON inference_endpoints(protocol);
+
+-- Auto-update updated_at on inference_endpoints changes
+DROP TRIGGER IF EXISTS trg_inference_endpoints_updated_at ON inference_endpoints;
+CREATE TRIGGER trg_inference_endpoints_updated_at
+    BEFORE UPDATE ON inference_endpoints
+    FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
+
+-- ─── model_deployments ────────────────────────────────────────────────────────
+-- A model served by an endpoint, with declared capabilities and limits.
+CREATE TABLE IF NOT EXISTS model_deployments (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    endpoint_id UUID NOT NULL REFERENCES inference_endpoints(id) ON DELETE CASCADE,
+    served_model_name TEXT NOT NULL,
+    display_name TEXT NOT NULL,
+    capabilities JSONB NOT NULL,
+    context_window INTEGER,
+    max_output_tokens INTEGER,
+    quantization TEXT,
+    runtime_metadata JSONB NOT NULL DEFAULT '{}',
+    enabled BOOLEAN NOT NULL DEFAULT TRUE,
+    revision INTEGER NOT NULL DEFAULT 1,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    UNIQUE(endpoint_id, served_model_name)
+);
+
+CREATE INDEX IF NOT EXISTS idx_model_deployments_endpoint
+    ON model_deployments(endpoint_id);
+
+-- Auto-update updated_at on model_deployments changes
+DROP TRIGGER IF EXISTS trg_model_deployments_updated_at ON model_deployments;
+CREATE TRIGGER trg_model_deployments_updated_at
+    BEFORE UPDATE ON model_deployments
+    FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
+
+-- ─── agent_stage_bindings ─────────────────────────────────────────────────────
+-- Maps an agent + pipeline stage to one or more ordered model deployments.
+CREATE TABLE IF NOT EXISTS agent_stage_bindings (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    agent_id UUID NOT NULL REFERENCES ai_agents(id) ON DELETE CASCADE,
+    stage TEXT NOT NULL,
+    model_deployment_id UUID REFERENCES model_deployments(id) ON DELETE SET NULL,
+    route_order INTEGER NOT NULL DEFAULT 0,
+    routing_config JSONB NOT NULL DEFAULT '{}',
+    is_active BOOLEAN NOT NULL DEFAULT TRUE,
+    revision INTEGER NOT NULL DEFAULT 1,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    UNIQUE(agent_id, stage, route_order)
+);
+
+CREATE INDEX IF NOT EXISTS idx_agent_stage_bindings_agent
+    ON agent_stage_bindings(agent_id);
+
+CREATE INDEX IF NOT EXISTS idx_agent_stage_bindings_deployment
+    ON agent_stage_bindings(model_deployment_id);
+
+-- Auto-update updated_at on agent_stage_bindings changes
+DROP TRIGGER IF EXISTS trg_agent_stage_bindings_updated_at ON agent_stage_bindings;
+CREATE TRIGGER trg_agent_stage_bindings_updated_at
+    BEFORE UPDATE ON agent_stage_bindings
+    FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
+
+-- ─── Additive lineage columns on agent_performance_log ────────────────────────
+-- Tracks which endpoint/deployment/binding was used for each logged invocation.
+-- NOTE: Revision increment logic is handled at the application layer:
+--   each UPDATE to inference_endpoints, model_deployments, or agent_stage_bindings
+--   should increment the revision column (enforced by service code, not DB trigger,
+--   to allow flexible conflict resolution).
+
+ALTER TABLE agent_performance_log
+    ADD COLUMN IF NOT EXISTS endpoint_id UUID REFERENCES inference_endpoints(id) ON DELETE SET NULL;
+
+ALTER TABLE agent_performance_log
+    ADD COLUMN IF NOT EXISTS deployment_id UUID REFERENCES model_deployments(id) ON DELETE SET NULL;
+
+ALTER TABLE agent_performance_log
+    ADD COLUMN IF NOT EXISTS binding_revision INTEGER;
+
+ALTER TABLE agent_performance_log
+    ADD COLUMN IF NOT EXISTS structured_mode TEXT;
+
+CREATE INDEX IF NOT EXISTS idx_agent_perf_endpoint
+    ON agent_performance_log(endpoint_id)
+    WHERE endpoint_id IS NOT NULL;
+
+CREATE INDEX IF NOT EXISTS idx_agent_perf_deployment
+    ON agent_performance_log(deployment_id)
+    WHERE deployment_id IS NOT NULL;

infra/migrations/041_v3_pipeline_tables.sql

@@ -0,0 +1,350 @@
+-- Migration 041: V3 Pipeline Persistence Tables
+-- Creates tables for the Intelligence Pipeline v3 staged evidence architecture:
+--   v3_pipeline_runs, v3_stage_runs, v3_document_chunks, v3_evidence_spans,
+--   v3_extracted_entities, v3_extracted_facts, v3_extracted_relations,
+--   v3_rejected_candidates, v3_company_signal_candidates,
+--   v3_adjudication_decisions, v3_routing_decisions, v3_stage_lineage
+-- Includes idempotency keys, immutable-revision constraints, and appropriate indexes.
+
+-- ═══════════════════════════════════════════════════════════════════════════════
+-- 20.1: Pipeline runs and stage runs
+-- ═══════════════════════════════════════════════════════════════════════════════
+
+-- ─── v3_pipeline_runs ─────────────────────────────────────────────────────────
+-- Top-level pipeline execution record for a document.
+CREATE TABLE IF NOT EXISTS v3_pipeline_runs (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    document_id UUID NOT NULL,
+    pipeline_version TEXT NOT NULL DEFAULT 'v3.0',
+    status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'running', 'completed', 'failed')),
+    started_at TIMESTAMPTZ,
+    completed_at TIMESTAMPTZ,
+    idempotency_key TEXT NOT NULL UNIQUE,
+    error TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_pipeline_runs_document
+    ON v3_pipeline_runs(document_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_pipeline_runs_status
+    ON v3_pipeline_runs(status);
+
+CREATE INDEX IF NOT EXISTS idx_v3_pipeline_runs_created
+    ON v3_pipeline_runs(created_at DESC);
+
+-- ─── v3_stage_runs ────────────────────────────────────────────────────────────
+-- Individual stage execution within a pipeline run.
+CREATE TABLE IF NOT EXISTS v3_stage_runs (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    pipeline_run_id UUID NOT NULL REFERENCES v3_pipeline_runs(id) ON DELETE CASCADE,
+    stage TEXT NOT NULL CHECK (stage IN (
+        'segmentation', 'extraction', 'sentiment', 'novelty',
+        'routing', 'adjudication', 'impact', 'persistence'
+    )),
+    status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'running', 'completed', 'failed', 'skipped')),
+    started_at TIMESTAMPTZ,
+    completed_at TIMESTAMPTZ,
+    endpoint_id UUID REFERENCES inference_endpoints(id) ON DELETE SET NULL,
+    deployment_id UUID REFERENCES model_deployments(id) ON DELETE SET NULL,
+    model_version TEXT,
+    schema_version TEXT,
+    input_refs JSONB NOT NULL DEFAULT '[]',
+    output_refs JSONB NOT NULL DEFAULT '[]',
+    trace_id TEXT,
+    error TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_stage_runs_pipeline
+    ON v3_stage_runs(pipeline_run_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_stage_runs_stage
+    ON v3_stage_runs(stage);
+
+CREATE INDEX IF NOT EXISTS idx_v3_stage_runs_status
+    ON v3_stage_runs(status);
+
+-- ═══════════════════════════════════════════════════════════════════════════════
+-- 20.2: Document chunks and evidence spans
+-- ═══════════════════════════════════════════════════════════════════════════════
+
+-- ─── v3_document_chunks ───────────────────────────────────────────────────────
+-- Segmented document chunks with offset tracking.
+CREATE TABLE IF NOT EXISTS v3_document_chunks (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    document_id UUID NOT NULL,
+    chunk_id TEXT NOT NULL,
+    section_path JSONB NOT NULL DEFAULT '[]',
+    speaker TEXT,
+    start_char INTEGER NOT NULL,
+    end_char INTEGER NOT NULL,
+    text TEXT NOT NULL,
+    overlap_left INTEGER NOT NULL DEFAULT 0,
+    overlap_right INTEGER NOT NULL DEFAULT 0,
+    boilerplate_score REAL NOT NULL DEFAULT 0.0,
+    document_type TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    UNIQUE(document_id, chunk_id)
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_document_chunks_document
+    ON v3_document_chunks(document_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_document_chunks_type
+    ON v3_document_chunks(document_type)
+    WHERE document_type IS NOT NULL;
+
+-- ─── v3_evidence_spans ────────────────────────────────────────────────────────
+-- Exact source text with character offsets for provenance.
+CREATE TABLE IF NOT EXISTS v3_evidence_spans (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    document_id UUID NOT NULL,
+    chunk_id TEXT,
+    start_char INTEGER NOT NULL,
+    end_char INTEGER NOT NULL,
+    text TEXT NOT NULL,
+    checksum TEXT NOT NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_evidence_spans_document
+    ON v3_evidence_spans(document_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_evidence_spans_checksum
+    ON v3_evidence_spans(checksum);
+
+-- ═══════════════════════════════════════════════════════════════════════════════
+-- 20.3: Extracted entities, facts, relations, and rejected candidates
+-- ═══════════════════════════════════════════════════════════════════════════════
+
+-- ─── v3_extracted_entities ────────────────────────────────────────────────────
+-- Entities discovered during extraction (companies, people, orgs, etc.).
+CREATE TABLE IF NOT EXISTS v3_extracted_entities (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    pipeline_run_id UUID NOT NULL REFERENCES v3_pipeline_runs(id) ON DELETE CASCADE,
+    entity_type TEXT NOT NULL,
+    literal_text TEXT NOT NULL,
+    canonical_id UUID,
+    evidence_span_id UUID REFERENCES v3_evidence_spans(id) ON DELETE SET NULL,
+    confidence REAL NOT NULL DEFAULT 0.0,
+    derivation TEXT NOT NULL DEFAULT 'specialist',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_entities_pipeline
+    ON v3_extracted_entities(pipeline_run_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_entities_canonical
+    ON v3_extracted_entities(canonical_id)
+    WHERE canonical_id IS NOT NULL;
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_entities_type
+    ON v3_extracted_entities(entity_type);
+
+-- ─── v3_extracted_facts ───────────────────────────────────────────────────────
+-- Structured facts (numeric values, dates, amounts, etc.).
+CREATE TABLE IF NOT EXISTS v3_extracted_facts (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    pipeline_run_id UUID NOT NULL REFERENCES v3_pipeline_runs(id) ON DELETE CASCADE,
+    fact_type TEXT NOT NULL,
+    subject_entity_id UUID REFERENCES v3_extracted_entities(id) ON DELETE SET NULL,
+    predicate TEXT NOT NULL,
+    literal_value TEXT NOT NULL,
+    normalized_value JSONB,
+    unit TEXT,
+    period JSONB,
+    evidence_span_ids UUID[] NOT NULL DEFAULT '{}',
+    confidence REAL NOT NULL DEFAULT 0.0,
+    derivation TEXT NOT NULL DEFAULT 'deterministic',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_facts_pipeline
+    ON v3_extracted_facts(pipeline_run_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_facts_subject
+    ON v3_extracted_facts(subject_entity_id)
+    WHERE subject_entity_id IS NOT NULL;
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_facts_type
+    ON v3_extracted_facts(fact_type);
+
+-- ─── v3_extracted_relations ───────────────────────────────────────────────────
+-- Relations between entities (competes_with, supplies, etc.).
+CREATE TABLE IF NOT EXISTS v3_extracted_relations (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    pipeline_run_id UUID NOT NULL REFERENCES v3_pipeline_runs(id) ON DELETE CASCADE,
+    relation_type TEXT NOT NULL,
+    source_entity_id UUID NOT NULL REFERENCES v3_extracted_entities(id) ON DELETE CASCADE,
+    target_entity_id UUID NOT NULL REFERENCES v3_extracted_entities(id) ON DELETE CASCADE,
+    evidence_span_ids UUID[] NOT NULL DEFAULT '{}',
+    confidence REAL NOT NULL DEFAULT 0.0,
+    derivation TEXT NOT NULL DEFAULT 'specialist',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_relations_pipeline
+    ON v3_extracted_relations(pipeline_run_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_relations_source
+    ON v3_extracted_relations(source_entity_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_extracted_relations_target
+    ON v3_extracted_relations(target_entity_id);
+
+-- ─── v3_rejected_candidates ──────────────────────────────────────────────────
+-- Candidates that failed validation or were rejected by a stage.
+CREATE TABLE IF NOT EXISTS v3_rejected_candidates (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    pipeline_run_id UUID NOT NULL REFERENCES v3_pipeline_runs(id) ON DELETE CASCADE,
+    candidate_type TEXT NOT NULL,
+    candidate_data JSONB NOT NULL,
+    rejection_reason TEXT NOT NULL,
+    stage TEXT NOT NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_rejected_candidates_pipeline
+    ON v3_rejected_candidates(pipeline_run_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_rejected_candidates_stage
+    ON v3_rejected_candidates(stage);
+
+-- ═══════════════════════════════════════════════════════════════════════════════
+-- 20.4: Company signal candidates and probability distributions
+-- ═══════════════════════════════════════════════════════════════════════════════
+
+-- ─── v3_company_signal_candidates ─────────────────────────────────────────────
+-- Per-company signal output with full probability distributions.
+CREATE TABLE IF NOT EXISTS v3_company_signal_candidates (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    pipeline_run_id UUID NOT NULL REFERENCES v3_pipeline_runs(id) ON DELETE CASCADE,
+    company_id UUID NOT NULL REFERENCES companies(id) ON DELETE CASCADE,
+    relevance_probability REAL NOT NULL DEFAULT 0.0,
+    event_probabilities JSONB NOT NULL DEFAULT '{}',
+    sentiment_probabilities JSONB NOT NULL DEFAULT '{}',
+    direction_probabilities JSONB NOT NULL DEFAULT '{}',
+    horizon_probabilities JSONB NOT NULL DEFAULT '{}',
+    expected_magnitude REAL,
+    evidence_span_ids UUID[] NOT NULL DEFAULT '{}',
+    routing_reasons TEXT[] NOT NULL DEFAULT '{}',
+    adjudicated BOOLEAN NOT NULL DEFAULT FALSE,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_signal_candidates_pipeline
+    ON v3_company_signal_candidates(pipeline_run_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_signal_candidates_company
+    ON v3_company_signal_candidates(company_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_signal_candidates_adjudicated
+    ON v3_company_signal_candidates(adjudicated)
+    WHERE adjudicated = TRUE;
+
+-- ═══════════════════════════════════════════════════════════════════════════════
+-- 20.5: Adjudication decisions, routing reasons, calibration, and model lineage
+-- ═══════════════════════════════════════════════════════════════════════════════
+
+-- ─── v3_stage_lineage ─────────────────────────────────────────────────────────
+-- Detailed model/endpoint lineage for each stage invocation.
+-- Created before adjudication_decisions because it is referenced as a FK.
+CREATE TABLE IF NOT EXISTS v3_stage_lineage (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    stage_run_id UUID NOT NULL REFERENCES v3_stage_runs(id) ON DELETE CASCADE,
+    endpoint_id UUID REFERENCES inference_endpoints(id) ON DELETE SET NULL,
+    deployment_id UUID REFERENCES model_deployments(id) ON DELETE SET NULL,
+    model TEXT,
+    protocol TEXT,
+    structured_mode TEXT,
+    request_id TEXT,
+    latency_ms INTEGER,
+    retries INTEGER NOT NULL DEFAULT 0,
+    trace_id TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_stage_lineage_stage_run
+    ON v3_stage_lineage(stage_run_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_stage_lineage_endpoint
+    ON v3_stage_lineage(endpoint_id)
+    WHERE endpoint_id IS NOT NULL;
+
+-- ─── v3_adjudication_decisions ────────────────────────────────────────────────
+-- Decisions made by the 9B adjudicator for ambiguous documents.
+CREATE TABLE IF NOT EXISTS v3_adjudication_decisions (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    pipeline_run_id UUID NOT NULL REFERENCES v3_pipeline_runs(id) ON DELETE CASCADE,
+    question_codes TEXT[] NOT NULL DEFAULT '{}',
+    candidates JSONB NOT NULL DEFAULT '{}',
+    decision JSONB NOT NULL DEFAULT '{}',
+    evidence_span_ids UUID[] NOT NULL DEFAULT '{}',
+    model_lineage_id UUID REFERENCES v3_stage_lineage(id) ON DELETE SET NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_adjudication_pipeline
+    ON v3_adjudication_decisions(pipeline_run_id);
+
+-- ─── v3_routing_decisions ─────────────────────────────────────────────────────
+-- Records of fast-path vs adjudication routing decisions.
+CREATE TABLE IF NOT EXISTS v3_routing_decisions (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    pipeline_run_id UUID NOT NULL REFERENCES v3_pipeline_runs(id) ON DELETE CASCADE,
+    document_id UUID NOT NULL,
+    route TEXT NOT NULL CHECK (route IN ('fast_path', 'adjudication')),
+    reason_codes TEXT[] NOT NULL DEFAULT '{}',
+    confidence_features JSONB NOT NULL DEFAULT '{}',
+    decided_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_v3_routing_pipeline
+    ON v3_routing_decisions(pipeline_run_id);
+
+CREATE INDEX IF NOT EXISTS idx_v3_routing_route
+    ON v3_routing_decisions(route);
+
+-- ═══════════════════════════════════════════════════════════════════════════════
+-- 20.6: Idempotency and immutable-revision constraints
+-- ═══════════════════════════════════════════════════════════════════════════════
+
+-- Pipeline runs: idempotency_key UNIQUE is already defined above in the table.
+-- Stage runs: unique per pipeline_run_id + stage to prevent duplicate stage execution.
+CREATE UNIQUE INDEX IF NOT EXISTS idx_v3_stage_runs_idempotent
+    ON v3_stage_runs(pipeline_run_id, stage);
+
+-- Company signal candidates: unique per pipeline_run_id + company_id.
+CREATE UNIQUE INDEX IF NOT EXISTS idx_v3_signal_candidates_idempotent
+    ON v3_company_signal_candidates(pipeline_run_id, company_id);
+
+-- Routing decisions: unique per pipeline_run_id (one routing decision per run).
+CREATE UNIQUE INDEX IF NOT EXISTS idx_v3_routing_idempotent
+    ON v3_routing_decisions(pipeline_run_id);
+
+-- Evidence spans: unique by document + checksum to avoid storing duplicates.
+CREATE UNIQUE INDEX IF NOT EXISTS idx_v3_evidence_spans_idempotent
+    ON v3_evidence_spans(document_id, checksum);
+
+-- Immutable revision rule: pipeline_runs and stage_runs cannot be updated once completed.
+-- Enforced via trigger: reject updates to rows where status = 'completed' or 'failed'.
+CREATE OR REPLACE FUNCTION v3_immutable_completed_row()
+RETURNS TRIGGER AS $$
+BEGIN
+    IF OLD.status IN ('completed', 'failed') THEN
+        RAISE EXCEPTION 'Cannot modify a % record with status=%', TG_TABLE_NAME, OLD.status;
+    END IF;
+    RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+DROP TRIGGER IF EXISTS trg_v3_pipeline_runs_immutable ON v3_pipeline_runs;
+CREATE TRIGGER trg_v3_pipeline_runs_immutable
+    BEFORE UPDATE ON v3_pipeline_runs
+    FOR EACH ROW EXECUTE FUNCTION v3_immutable_completed_row();
+
+DROP TRIGGER IF EXISTS trg_v3_stage_runs_immutable ON v3_stage_runs;
+CREATE TRIGGER trg_v3_stage_runs_immutable
+    BEFORE UPDATE ON v3_stage_runs
+    FOR EACH ROW EXECUTE FUNCTION v3_immutable_completed_row();

infra/migrations/042_seed_inference_registry.sql

@@ -0,0 +1,74 @@
+-- Migration 042: Seed Inference Registry
+-- Populates initial endpoint profiles and model deployments for the
+-- existing Ollama and vLLM services.
+--
+-- Task 18.1: Create the current Ollama endpoint profile
+-- Task 18.2: Create the current vLLM OpenAI-compatible endpoint profile
+-- Task 18.3: Create model deployments matching actual runtime state
+--
+-- This is a DATA migration. The schema was created in 040_inference_registry.sql.
+-- Uses ON CONFLICT DO NOTHING for idempotency.
+
+-- ─── 18.1: Ollama endpoint profile ───────────────────────────────────────────
+INSERT INTO inference_endpoints (id, name, protocol, base_url, auth_secret_ref, auth_scheme, default_headers, health_path, enabled)
+VALUES (
+    'a0000000-0000-4000-8000-000000000001'::uuid,
+    'stonks-ollama',
+    'ollama_native',
+    'http://ollama.ollama-service.svc.cluster.local:11434',
+    NULL,
+    'none',
+    '{}',
+    '/api/tags',
+    TRUE
+)
+ON CONFLICT (name) DO NOTHING;
+
+-- ─── 18.2: vLLM OpenAI-compatible endpoint profile ──────────────────────────
+INSERT INTO inference_endpoints (id, name, protocol, base_url, auth_secret_ref, auth_scheme, default_headers, health_path, enabled)
+VALUES (
+    'a0000000-0000-4000-8000-000000000002'::uuid,
+    'stonks-vllm',
+    'openai_chat',
+    'http://kube-vllm.stonks-oracle.svc.cluster.local:8000',
+    NULL,
+    'none',
+    '{}',
+    '/health',
+    TRUE
+)
+ON CONFLICT (name) DO NOTHING;
+
+-- ─── 18.3: Model deployments ─────────────────────────────────────────────────
+
+-- Ollama model deployment (qwen3.5:9b served via Ollama native protocol)
+INSERT INTO model_deployments (id, endpoint_id, served_model_name, display_name, capabilities, context_window, max_output_tokens, quantization, runtime_metadata, enabled)
+VALUES (
+    'b0000000-0000-4000-8000-000000000001'::uuid,
+    'a0000000-0000-4000-8000-000000000001'::uuid,
+    'qwen3.5:9b',
+    'Qwen 3.5 9B (Ollama)',
+    '{"chat_completions": true, "json_schema": false, "json_object": true, "seed": false, "usage": false, "max_completion_tokens": false, "model_listing": true}',
+    32768,
+    32768,
+    NULL,
+    '{"source": "ollama_native", "notes": "Ollama-served model with native JSON mode"}',
+    TRUE
+)
+ON CONFLICT (endpoint_id, served_model_name) DO NOTHING;
+
+-- vLLM model deployment (AxionML/Qwen3.5-9B-NVFP4 on RTX 4070 Ti SUPER)
+INSERT INTO model_deployments (id, endpoint_id, served_model_name, display_name, capabilities, context_window, max_output_tokens, quantization, runtime_metadata, enabled)
+VALUES (
+    'b0000000-0000-4000-8000-000000000002'::uuid,
+    'a0000000-0000-4000-8000-000000000002'::uuid,
+    'AxionML/Qwen3.5-9B-NVFP4',
+    'Qwen 3.5 9B NVFP4 (vLLM)',
+    '{"chat_completions": true, "json_schema": true, "json_object": true, "seed": true, "usage": true, "max_completion_tokens": true, "model_listing": true}',
+    8192,
+    2048,
+    'NVFP4',
+    '{"gpu": "RTX 4070 Ti SUPER", "gpu_memory_utilization": 0.80, "max_num_seqs": 8, "vllm_structured_outputs": true}',
+    TRUE
+)
+ON CONFLICT (endpoint_id, served_model_name) DO NOTHING;

requirements.txt

@@ -16,6 +16,9 @@ httpx>=0.27.0
 # JSON repair for LLM output
 json-repair>=0.59.0
 
+# JSON Schema validation
+jsonschema>=4.20.0
+
 # Web scraping
 beautifulsoup4>=4.12.0
 requests>=2.31.0
@@ -37,5 +40,9 @@ pyyaml>=6.0.0
 pytest>=8.0.0
 pytest-asyncio>=0.24.0
 
+# ML/Calibration (used by confidence and sentiment calibrators)
+numpy>=1.26.0
+scikit-learn>=1.4.0
+
 # Code quality
 ruff==0.15.10

scripts/backfill_snapshot_prices.py

@@ -0,0 +1,171 @@
+"""Backfill price_at_prediction for existing NULL prediction snapshots.
+
+One-time migration script that populates price_at_prediction using the
+extended fallback chain:
+  1. market_snapshots within 24h of generated_at for the ticker
+  2. positions table (current_price) for the ticker
+
+Run as: .venv/bin/python scripts/backfill_snapshot_prices.py
+Dry run: .venv/bin/python scripts/backfill_snapshot_prices.py --dry-run
+
+Requires env vars: POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST,
+                   POSTGRES_PORT, POSTGRES_DB
+
+Requirements: 2.3
+"""
+
+import argparse
+import asyncio
+import os
+import sys
+from datetime import timedelta
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import asyncpg  # noqa: E402
+
+from services.shared.config import load_config  # noqa: E402
+
+# ---------------------------------------------------------------------------
+# SQL Queries
+# ---------------------------------------------------------------------------
+
+_FIND_NULL_SNAPSHOTS_SQL = """
+SELECT id, ticker, generated_at
+FROM prediction_snapshots
+WHERE price_at_prediction IS NULL
+ORDER BY generated_at DESC
+"""
+
+_MARKET_SNAPSHOT_FALLBACK_SQL = """
+SELECT (data->>'c')::float AS close
+FROM market_snapshots
+WHERE ticker = $1
+  AND snapshot_type = 'bar'
+  AND data->>'c' IS NOT NULL
+  AND captured_at >= $2
+  AND captured_at <= $3
+ORDER BY captured_at DESC
+LIMIT 1
+"""
+
+_POSITIONS_FALLBACK_SQL = """
+SELECT current_price
+FROM positions
+WHERE ticker = $1
+  AND current_price IS NOT NULL
+LIMIT 1
+"""
+
+_UPDATE_PRICE_SQL = """
+UPDATE prediction_snapshots
+SET price_at_prediction = $1
+WHERE id = $2
+"""
+
+
+# ---------------------------------------------------------------------------
+# Main backfill logic
+# ---------------------------------------------------------------------------
+
+
+async def backfill(dry_run: bool = False) -> None:
+    config = load_config()
+    dsn = config.postgres.dsn
+
+    pool = await asyncpg.create_pool(dsn=dsn)
+    assert pool is not None
+
+    # Find all snapshots with NULL price
+    rows = await pool.fetch(_FIND_NULL_SNAPSHOTS_SQL)
+    total = len(rows)
+
+    if total == 0:
+        print("No prediction snapshots with NULL price_at_prediction found.")
+        await pool.close()
+        return
+
+    print(f"Found {total} snapshots with NULL price_at_prediction")
+    if dry_run:
+        print("[DRY RUN] No updates will be performed")
+    print()
+
+    # Statistics
+    found_market = 0
+    found_positions = 0
+    still_null = 0
+
+    for idx, row in enumerate(rows, start=1):
+        snapshot_id = row["id"]
+        ticker = row["ticker"]
+        generated_at = row["generated_at"]
+
+        price: float | None = None
+
+        # Fallback 1: market_snapshots within 24h of generated_at
+        window_start = generated_at - timedelta(hours=24)
+        market_row = await pool.fetchrow(
+            _MARKET_SNAPSHOT_FALLBACK_SQL, ticker, window_start, generated_at
+        )
+        if market_row and market_row["close"] is not None:
+            price = float(market_row["close"])
+            found_market += 1
+        else:
+            # Fallback 2: positions table
+            pos_row = await pool.fetchrow(_POSITIONS_FALLBACK_SQL, ticker)
+            if pos_row and pos_row["current_price"] is not None:
+                price = float(pos_row["current_price"])
+                found_positions += 1
+            else:
+                still_null += 1
+
+        # Update if we found a price
+        if price is not None and not dry_run:
+            await pool.execute(_UPDATE_PRICE_SQL, price, snapshot_id)
+
+        # Progress reporting every 100 snapshots
+        if idx % 100 == 0:
+            action = "checked" if dry_run else "processed"
+            print(
+                f"  {action} {idx}/{total} snapshots "
+                f"(market: {found_market}, positions: {found_positions}, "
+                f"null: {still_null})"
+            )
+
+    await pool.close()
+
+    # Final statistics
+    print()
+    print("=" * 60)
+    print("Backfill complete" if not dry_run else "Dry run complete")
+    print("=" * 60)
+    print(f"  Total snapshots processed: {total}")
+    print(f"  Found via market_snapshots: {found_market}")
+    print(f"  Found via positions:        {found_positions}")
+    print(f"  Still NULL (no data):       {still_null}")
+    if dry_run:
+        updated = found_market + found_positions
+        print(f"\n  [DRY RUN] Would have updated {updated} snapshots")
+    print()
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Backfill price_at_prediction for NULL prediction snapshots"
+    )
+    parser.add_argument(
+        "--dry-run",
+        action="store_true",
+        help="Report what would be updated without making changes",
+    )
+    args = parser.parse_args()
+
+    try:
+        asyncio.run(backfill(dry_run=args.dry_run))
+    except Exception as e:
+        print(f"Backfill failed: {e}", file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

scripts/pipeline_health_check.sh

@@ -0,0 +1,506 @@
+#!/usr/bin/env bash
+# Stonks Oracle — Deep Pipeline Health Check
+# Verifies every component is ACTUALLY working, not just "running"
+#
+# Usage: ./scripts/pipeline_health_check.sh [namespace]
+# Default namespace: stonks-oracle
+
+set -uo pipefail
+
+NS="${1:-stonks-oracle}"
+PASS=0
+WARN=0
+FAIL=0
+DETAILS=""
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+
+pass() { ((PASS++)); echo -e "  ${GREEN}✓${NC} $1"; }
+warn() { ((WARN++)); echo -e "  ${YELLOW}⚠${NC} $1"; }
+fail() { ((FAIL++)); echo -e "  ${RED}✗${NC} $1"; }
+header() { echo -e "\n${BLUE}━━━ $1 ━━━${NC}"; }
+
+# ─── 1. POD HEALTH ─────────────────────────────────────────────
+header "Pod Health (namespace: $NS)"
+
+# Check all deployments are at desired replicas
+while IFS= read -r line; do
+  name=$(echo "$line" | awk '{print $1}')
+  ready=$(echo "$line" | awk '{print $2}')
+  desired=$(echo "$ready" | cut -d/ -f2)
+  actual=$(echo "$ready" | cut -d/ -f1)
+  if [[ "$desired" == "0" ]]; then
+    continue  # intentionally scaled down
+  fi
+  if [[ "$actual" == "$desired" ]]; then
+    pass "$name: $ready"
+  else
+    fail "$name: $ready (not fully ready)"
+  fi
+done < <(kubectl get deployments -n "$NS" --no-headers 2>/dev/null)
+
+# Check for restarts in the last hour
+RESTART_PODS=$(kubectl get pods -n "$NS" --no-headers -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount 2>/dev/null | awk '$2 > 0 {print $1 "(" $2 ")"}')
+if [[ -z "$RESTART_PODS" ]]; then
+  pass "No pod restarts"
+else
+  for p in $RESTART_PODS; do
+    warn "Restart detected: $p"
+  done
+fi
+
+# Check for OOMKilled in last state
+OOM_PODS=$(kubectl get pods -n "$NS" -o json 2>/dev/null | python3 -c "
+import json, sys
+data = json.load(sys.stdin)
+for pod in data.get('items', []):
+    for cs in pod.get('status', {}).get('containerStatuses', []):
+        last = cs.get('lastState', {}).get('terminated', {})
+        if last.get('reason') == 'OOMKilled':
+            print(pod['metadata']['name'])
+" 2>/dev/null)
+if [[ -z "$OOM_PODS" ]]; then
+  pass "No OOMKilled containers"
+else
+  for p in $OOM_PODS; do
+    fail "OOMKilled: $p"
+  done
+fi
+
+# ─── 2. DATABASE CONNECTIVITY & HEALTH ────────────────────────
+header "PostgreSQL"
+
+DB_CHECK=$(kubectl exec -n "$NS" deployment/query-api -- python3 -c "
+import asyncio, asyncpg, os, json
+async def check():
+    pool = await asyncpg.create_pool(dsn=f\"postgresql://{os.environ['POSTGRES_USER']}:{os.environ['POSTGRES_PASSWORD']}@{os.environ['POSTGRES_HOST']}:{os.environ.get('POSTGRES_PORT','5432')}/{os.environ['POSTGRES_DB']}\", min_size=1, max_size=2)
+    result = {}
+    result['connected'] = True
+    # DB size
+    row = await pool.fetchrow('SELECT pg_database_size(current_database()) as size')
+    result['db_size_mb'] = round(row['size'] / 1024 / 1024)
+    # Table count
+    result['table_count'] = await pool.fetchval(\"SELECT count(*) FROM pg_tables WHERE schemaname='public'\")
+    # Active connections
+    result['connections'] = await pool.fetchval('SELECT count(*) FROM pg_stat_activity WHERE datname = current_database()')
+    await pool.close()
+    print(json.dumps(result))
+asyncio.run(check())
+" 2>/dev/null) || DB_CHECK='{"connected": false}'
+
+if echo "$DB_CHECK" | python3 -c "import json,sys; d=json.load(sys.stdin); exit(0 if d.get('connected') else 1)" 2>/dev/null; then
+  DB_SIZE=$(echo "$DB_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin)['db_size_mb'])")
+  DB_TABLES=$(echo "$DB_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin)['table_count'])")
+  DB_CONNS=$(echo "$DB_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin)['connections'])")
+  pass "Connected (${DB_SIZE}MB, ${DB_TABLES} tables, ${DB_CONNS} connections)"
+else
+  fail "Cannot connect to PostgreSQL"
+fi
+
+# ─── 3. REDIS CONNECTIVITY & QUEUES ───────────────────────────
+header "Redis"
+
+REDIS_CHECK=$(kubectl exec -n "$NS" deployment/query-api -- python3 -c "
+import redis, os, json
+r = redis.from_url(f'redis://:{os.environ.get(\"REDIS_PASSWORD\",\"\")}@{os.environ.get(\"REDIS_HOST\",\"redis-master.redis-service.svc.cluster.local\")}:6379/{os.environ.get(\"REDIS_DB\",\"1\")}')
+result = {}
+result['connected'] = True
+result['memory'] = r.info('memory')['used_memory_human']
+result['keys'] = r.dbsize()
+# Check queues for backlog
+queues = {}
+for q in ['ingestion','parsing','extraction','aggregation','recommendation','trading','broker_orders','macro_classification']:
+    stage = os.environ.get('DEPLOY_STAGE', '')
+    prefix = f'stonks:{stage}' if stage else 'stonks'
+    key = f'{prefix}:queue:{q}'
+    length = r.llen(key)
+    if length > 0:
+        queues[q] = length
+result['queue_backlog'] = queues
+print(json.dumps(result))
+" 2>/dev/null) || REDIS_CHECK='{"connected": false}'
+
+if echo "$REDIS_CHECK" | python3 -c "import json,sys; d=json.load(sys.stdin); exit(0 if d.get('connected') else 1)" 2>/dev/null; then
+  R_MEM=$(echo "$REDIS_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin)['memory'])")
+  R_KEYS=$(echo "$REDIS_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin)['keys'])")
+  pass "Connected (${R_MEM}, ${R_KEYS} keys)"
+  BACKLOG=$(echo "$REDIS_CHECK" | python3 -c "import json,sys; d=json.load(sys.stdin)['queue_backlog']; print(' '.join(f'{k}={v}' for k,v in d.items()) if d else '')")
+  if [[ -z "$BACKLOG" ]]; then
+    pass "All queues drained (no backlog)"
+  else
+    warn "Queue backlog: $BACKLOG"
+  fi
+else
+  fail "Cannot connect to Redis"
+fi
+
+# ─── 4. MINIO CONNECTIVITY ────────────────────────────────────
+header "MinIO (S3)"
+
+MINIO_CHECK=$(kubectl exec -n "$NS" deployment/query-api -- python3 -c "
+from minio import Minio
+import os, json
+client = Minio(
+    os.environ.get('MINIO_ENDPOINT', 'minio.minio-service.svc.cluster.local:80'),
+    access_key=os.environ.get('MINIO_ACCESS_KEY', ''),
+    secret_key=os.environ.get('MINIO_SECRET_KEY', ''),
+    secure=False
+)
+buckets = client.list_buckets()
+print(json.dumps({'connected': True, 'bucket_count': len(buckets)}))
+" 2>/dev/null) || MINIO_CHECK='{"connected": false}'
+
+if echo "$MINIO_CHECK" | python3 -c "import json,sys; d=json.load(sys.stdin); exit(0 if d.get('connected') else 1)" 2>/dev/null; then
+  BUCKETS=$(echo "$MINIO_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin)['bucket_count'])")
+  pass "Connected ($BUCKETS buckets)"
+else
+  fail "Cannot connect to MinIO"
+fi
+
+# ─── 5. vLLM / AI MODEL ───────────────────────────────────────
+header "vLLM (NuExtract3)"
+
+VLLM_CHECK=$(kubectl exec -n "$NS" deployment/extractor -- python3 -c "
+import httpx, json, os
+url = os.environ.get('VLLM_BASE_URL', os.environ.get('OLLAMA_BASE_URL', ''))
+try:
+    resp = httpx.get(f'{url}/v1/models', timeout=10)
+    models = resp.json().get('data', [])
+    names = [m['id'] for m in models]
+    print(json.dumps({'connected': True, 'models': names}))
+except Exception as e:
+    print(json.dumps({'connected': False, 'error': str(e)}))
+" 2>/dev/null) || VLLM_CHECK='{"connected": false}'
+
+if echo "$VLLM_CHECK" | python3 -c "import json,sys; d=json.load(sys.stdin); exit(0 if d.get('connected') else 1)" 2>/dev/null; then
+  MODELS=$(echo "$VLLM_CHECK" | python3 -c "import json,sys; print(', '.join(json.load(sys.stdin)['models']))")
+  pass "Connected (models: $MODELS)"
+else
+  fail "Cannot reach vLLM"
+fi
+
+# ─── 6. BROKER CONNECTIVITY ───────────────────────────────────
+header "Alpaca Broker"
+
+BROKER_CHECK=$(kubectl exec -n "$NS" deployment/broker-adapter -- python3 -c "
+import httpx, json, os
+key = os.environ.get('BROKER_API_KEY', '')
+secret = os.environ.get('BROKER_API_SECRET', '')
+base = os.environ.get('BROKER_BASE_URL', 'https://paper-api.alpaca.markets')
+try:
+    resp = httpx.get(f'{base}/v2/account', headers={'APCA-API-KEY-ID': key, 'APCA-API-SECRET-KEY': secret}, timeout=10)
+    if resp.status_code == 200:
+        acct = resp.json()
+        print(json.dumps({'connected': True, 'equity': acct.get('equity'), 'buying_power': acct.get('buying_power'), 'status': acct.get('status')}))
+    else:
+        print(json.dumps({'connected': False, 'status_code': resp.status_code}))
+except Exception as e:
+    print(json.dumps({'connected': False, 'error': str(e)}))
+" 2>/dev/null) || BROKER_CHECK='{"connected": false}'
+
+if echo "$BROKER_CHECK" | python3 -c "import json,sys; d=json.load(sys.stdin); exit(0 if d.get('connected') else 1)" 2>/dev/null; then
+  EQUITY=$(echo "$BROKER_CHECK" | python3 -c "import json,sys; d=json.load(sys.stdin); print(f'equity=\${float(d[\"equity\"]):,.0f} buying_power=\${float(d[\"buying_power\"]):,.0f}')")
+  pass "Connected ($EQUITY)"
+else
+  fail "Cannot reach Alpaca API"
+fi
+
+# ─── 7. PIPELINE FLOW (the real test) ─────────────────────────
+header "Pipeline Flow (last 6 hours)"
+
+FLOW_CHECK=$(kubectl exec -n "$NS" deployment/query-api -- python3 -c "
+import asyncio, asyncpg, os, json
+from datetime import datetime, timezone
+async def check():
+    pool = await asyncpg.create_pool(dsn=f\"postgresql://{os.environ['POSTGRES_USER']}:{os.environ['POSTGRES_PASSWORD']}@{os.environ['POSTGRES_HOST']}:{os.environ.get('POSTGRES_PORT','5432')}/{os.environ['POSTGRES_DB']}\", min_size=1, max_size=2)
+    r = {}
+    now = datetime.now(tz=timezone.utc)
+    # Ingestion: new docs created
+    r['docs_ingested_6h'] = await pool.fetchval(\"SELECT count(*) FROM documents WHERE created_at > now() - interval '6 hours'\")
+    # Extraction: docs extracted recently
+    r['docs_extracted_6h'] = await pool.fetchval(\"SELECT count(*) FROM documents WHERE status = 'extracted' AND updated_at > now() - interval '6 hours'\")
+    # Stuck docs
+    r['stuck_parsed'] = await pool.fetchval(\"SELECT count(*) FROM documents WHERE status = 'parsed' AND updated_at < now() - interval '1 hour'\")
+    r['stuck_ingested'] = await pool.fetchval(\"SELECT count(*) FROM documents WHERE status = 'ingested' AND updated_at < now() - interval '1 hour'\")
+    # Aggregation: trend history entries
+    r['trends_6h'] = await pool.fetchval(\"SELECT count(*) FROM trend_history WHERE generated_at > now() - interval '6 hours'\")
+    # Recommendations generated
+    r['recs_6h'] = await pool.fetchval(\"SELECT count(*) FROM recommendations WHERE created_at > now() - interval '6 hours'\")
+    # Trading decisions
+    r['decisions_6h'] = await pool.fetchval(\"SELECT count(*) FROM trading_decisions WHERE created_at > now() - interval '6 hours'\")
+    # Global events (macro pipeline)
+    r['global_events_6h'] = await pool.fetchval(\"SELECT count(*) FROM global_events WHERE created_at > now() - interval '6 hours'\")
+    # Prediction snapshots
+    r['snapshots_6h'] = await pool.fetchval(\"SELECT count(*) FROM prediction_snapshots WHERE created_at > now() - interval '6 hours'\")
+    # Market data freshness
+    latest_market = await pool.fetchval('SELECT max(captured_at) FROM market_snapshots')
+    if latest_market:
+        r['market_data_age_min'] = round((now - latest_market).total_seconds() / 60)
+    else:
+        r['market_data_age_min'] = -1
+    await pool.close()
+    print(json.dumps(r))
+asyncio.run(check())
+" 2>/dev/null) || FLOW_CHECK='{}'
+
+if [[ -n "$FLOW_CHECK" && "$FLOW_CHECK" != "{}" ]]; then
+  DOCS_IN=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('docs_ingested_6h', 0))")
+  DOCS_EX=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('docs_extracted_6h', 0))")
+  STUCK_P=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('stuck_parsed', 0))")
+  STUCK_I=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('stuck_ingested', 0))")
+  TRENDS=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('trends_6h', 0))")
+  RECS=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('recs_6h', 0))")
+  DECISIONS=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('decisions_6h', 0))")
+  EVENTS=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('global_events_6h', 0))")
+  SNAPS=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('snapshots_6h', 0))")
+  MKT_AGE=$(echo "$FLOW_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('market_data_age_min', -1))")
+
+  # Ingestion flowing?
+  if [[ "$DOCS_IN" -gt 0 ]]; then
+    pass "Ingestion: $DOCS_IN docs ingested"
+  else
+    fail "Ingestion: ZERO docs in last 6h"
+  fi
+
+  # Extraction flowing?
+  if [[ "$DOCS_EX" -gt 0 ]]; then
+    pass "Extraction: $DOCS_EX docs extracted"
+  else
+    fail "Extraction: ZERO docs extracted in last 6h"
+  fi
+
+  # Stuck documents?
+  if [[ "$STUCK_P" -eq 0 && "$STUCK_I" -eq 0 ]]; then
+    pass "No stuck documents"
+  else
+    [[ "$STUCK_P" -gt 0 ]] && warn "Stuck in 'parsed': $STUCK_P docs (>1h old)"
+    [[ "$STUCK_I" -gt 0 ]] && warn "Stuck in 'ingested': $STUCK_I docs (>1h old)"
+  fi
+
+  # Aggregation producing trends?
+  if [[ "$TRENDS" -gt 0 ]]; then
+    pass "Aggregation: $TRENDS trend entries"
+  else
+    warn "Aggregation: 0 trends (expected on weekends/off-hours)"
+  fi
+
+  # Recommendations flowing?
+  if [[ "$RECS" -gt 0 ]]; then
+    pass "Recommendations: $RECS generated"
+  else
+    fail "Recommendations: ZERO in last 6h"
+  fi
+
+  # Trading decisions happening?
+  if [[ "$DECISIONS" -gt 0 ]]; then
+    pass "Trading engine: $DECISIONS decisions"
+  else
+    warn "Trading engine: 0 decisions (market may be closed)"
+  fi
+
+  # Macro events?
+  if [[ "$EVENTS" -gt 0 ]]; then
+    pass "Macro pipeline: $EVENTS global events classified"
+  else
+    warn "Macro pipeline: 0 events (may be low news volume)"
+  fi
+
+  # Prediction snapshots?
+  if [[ "$SNAPS" -gt 0 ]]; then
+    pass "Validation: $SNAPS prediction snapshots"
+  else
+    warn "Validation: 0 snapshots in 6h"
+  fi
+
+  # Market data freshness
+  if [[ "$MKT_AGE" -ge 0 && "$MKT_AGE" -lt 30 ]]; then
+    pass "Market data: ${MKT_AGE}min old"
+  elif [[ "$MKT_AGE" -ge 30 && "$MKT_AGE" -lt 120 ]]; then
+    warn "Market data: ${MKT_AGE}min old (stale)"
+  elif [[ "$MKT_AGE" -ge 120 ]]; then
+    # Could be weekend/after hours
+    HOURS=$((MKT_AGE / 60))
+    warn "Market data: ${HOURS}h old (weekend/after-hours?)"
+  fi
+else
+  fail "Could not query pipeline flow data"
+fi
+
+# ─── 8. QUALITY GATE STATUS ───────────────────────────────────
+header "Model Quality Gate"
+
+QG_CHECK=$(kubectl exec -n "$NS" deployment/query-api -- python3 -c "
+import asyncio, asyncpg, os, json
+from datetime import datetime, timezone
+async def check():
+    pool = await asyncpg.create_pool(dsn=f\"postgresql://{os.environ['POSTGRES_USER']}:{os.environ['POSTGRES_PASSWORD']}@{os.environ['POSTGRES_HOST']}:{os.environ.get('POSTGRES_PORT','5432')}/{os.environ['POSTGRES_DB']}\", min_size=1, max_size=2)
+    row = await pool.fetchrow(\"\"\"
+        SELECT generated_at, prediction_count, win_rate, directional_accuracy, brier_score
+        FROM model_metric_snapshots
+        WHERE lookback_window = '30d' AND horizon = '7d'
+        ORDER BY generated_at DESC LIMIT 1
+    \"\"\")
+    r = {}
+    if row:
+        age_h = (datetime.now(tz=timezone.utc) - row['generated_at']).total_seconds() / 3600
+        r['age_hours'] = round(age_h, 1)
+        r['prediction_count'] = row['prediction_count']
+        r['win_rate'] = float(row['win_rate']) if row['win_rate'] else 0
+        r['directional_accuracy'] = float(row['directional_accuracy']) if row['directional_accuracy'] else 0
+        r['brier_score'] = float(row['brier_score']) if row['brier_score'] else 0
+    else:
+        r['age_hours'] = -1
+    await pool.close()
+    print(json.dumps(r))
+asyncio.run(check())
+" 2>/dev/null) || QG_CHECK='{}'
+
+if [[ -n "$QG_CHECK" && "$QG_CHECK" != "{}" ]]; then
+  QG_AGE=$(echo "$QG_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('age_hours', -1))")
+  QG_PREDS=$(echo "$QG_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('prediction_count', 0))")
+  QG_WR=$(echo "$QG_CHECK" | python3 -c "import json,sys; print(f\"{json.load(sys.stdin).get('win_rate', 0)*100:.1f}%\")")
+  QG_DA=$(echo "$QG_CHECK" | python3 -c "import json,sys; print(f\"{json.load(sys.stdin).get('directional_accuracy', 0)*100:.1f}%\")")
+
+  if python3 -c "exit(0 if $QG_AGE >= 0 and $QG_AGE <= 48 else 1)" 2>/dev/null; then
+    pass "Metric snapshot: ${QG_AGE}h old (predictions=$QG_PREDS, win_rate=$QG_WR, accuracy=$QG_DA)"
+  elif python3 -c "exit(0 if $QG_AGE > 48 else 1)" 2>/dev/null; then
+    fail "Metric snapshot STALE: ${QG_AGE}h old (max 48h) — quality gate failing"
+  else
+    fail "No metric snapshot exists"
+  fi
+fi
+
+# ─── 9. PORTFOLIO STATUS ──────────────────────────────────────
+header "Paper Trading Portfolio"
+
+PORT_CHECK=$(kubectl exec -n "$NS" deployment/query-api -- python3 -c "
+import asyncio, asyncpg, os, json
+async def check():
+    pool = await asyncpg.create_pool(dsn=f\"postgresql://{os.environ['POSTGRES_USER']}:{os.environ['POSTGRES_PASSWORD']}@{os.environ['POSTGRES_HOST']}:{os.environ.get('POSTGRES_PORT','5432')}/{os.environ['POSTGRES_DB']}\", min_size=1, max_size=2)
+    r = {}
+    positions = await pool.fetch('SELECT ticker, quantity, avg_entry_price, current_price, unrealized_pnl FROM positions')
+    r['position_count'] = len(positions)
+    r['total_invested'] = sum(float(p['avg_entry_price']) * float(p['quantity']) for p in positions)
+    r['unrealized_pnl'] = sum(float(p['unrealized_pnl'] or 0) for p in positions)
+    r['tickers'] = [p['ticker'] for p in positions]
+    # Recent orders
+    r['orders_24h'] = await pool.fetchval(\"SELECT count(*) FROM orders WHERE created_at > now() - interval '24 hours'\")
+    r['total_orders'] = await pool.fetchval('SELECT count(*) FROM orders')
+    await pool.close()
+    print(json.dumps(r))
+asyncio.run(check())
+" 2>/dev/null) || PORT_CHECK='{}'
+
+if [[ -n "$PORT_CHECK" && "$PORT_CHECK" != "{}" ]]; then
+  POS_CNT=$(echo "$PORT_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('position_count', 0))")
+  INVESTED=$(echo "$PORT_CHECK" | python3 -c "import json,sys; print(f\"\${json.load(sys.stdin).get('total_invested', 0):,.0f}\")")
+  PNL=$(echo "$PORT_CHECK" | python3 -c "import json,sys; print(f\"\${json.load(sys.stdin).get('unrealized_pnl', 0):,.0f}\")")
+  TICKERS=$(echo "$PORT_CHECK" | python3 -c "import json,sys; print(', '.join(json.load(sys.stdin).get('tickers', [])))")
+  ORDERS_24=$(echo "$PORT_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('orders_24h', 0))")
+  TOTAL_ORD=$(echo "$PORT_CHECK" | python3 -c "import json,sys; print(json.load(sys.stdin).get('total_orders', 0))")
+  pass "$POS_CNT positions ($TICKERS)"
+  pass "Invested: $INVESTED | Unrealized P&L: $PNL"
+  pass "Orders: $ORDERS_24 today, $TOTAL_ORD lifetime"
+fi
+
+# ─── 10. API ENDPOINTS ────────────────────────────────────────
+header "API Endpoints"
+
+# Query API health
+QA_STATUS=$(kubectl exec -n "$NS" deployment/query-api -- curl -sf http://localhost:8000/health 2>/dev/null) && pass "Query API: healthy" || fail "Query API: unreachable"
+
+# Symbol Registry health
+SR_STATUS=$(kubectl exec -n "$NS" deployment/symbol-registry -- curl -sf http://localhost:8000/health 2>/dev/null) && pass "Symbol Registry: healthy" || fail "Symbol Registry: unreachable"
+
+# Trading Engine health
+TE_STATUS=$(kubectl exec -n "$NS" deployment/trading-engine -- curl -sf http://localhost:8000/health 2>/dev/null) && pass "Trading Engine: healthy" || fail "Trading Engine: unreachable"
+
+# Risk Engine health
+RISK_STATUS=$(kubectl exec -n "$NS" deployment/risk -- curl -sf http://localhost:8000/health 2>/dev/null) && pass "Risk Engine: healthy" || fail "Risk Engine: unreachable"
+
+# Specialist NER service health
+SPEC_STATUS=$(kubectl exec -n "$NS" deployment/specialist -- curl -sf http://localhost:8000/health 2>/dev/null) && pass "Specialist: healthy" || fail "Specialist: unreachable"
+
+# ─── 11. RESOURCE USAGE ───────────────────────────────────────
+header "Resource Usage"
+
+TOTAL_MEM=$(kubectl top pods -n "$NS" --no-headers 2>/dev/null | awk '{sum += $3} END {print sum}')
+TOTAL_CPU=$(kubectl top pods -n "$NS" --no-headers 2>/dev/null | awk '{sum += $2} END {print sum}')
+pass "Total: ${TOTAL_CPU}m CPU, ${TOTAL_MEM}Mi memory"
+
+# Any pod over 80% of its limit?
+HIGH_MEM=$(kubectl top pods -n "$NS" --no-headers 2>/dev/null | awk '$3+0 > 200 {print $1 "=" $3}')
+if [[ -n "$HIGH_MEM" ]]; then
+  warn "High memory pods: $HIGH_MEM"
+fi
+
+
+# ─── 12. END-TO-END LATENCY CHECK ─────────────────────────────
+header "End-to-End Pipeline Latency"
+
+# Copy helper script to pod and run it
+LATENCY_CHECK=$(kubectl exec -n "$NS" deployment/query-api -- python3 -c '
+import asyncio, asyncpg, os, json
+from datetime import datetime, timezone, timedelta
+async def check():
+    dsn = "postgresql://{}:{}@{}:{}/{}".format(os.environ["POSTGRES_USER"], os.environ["POSTGRES_PASSWORD"], os.environ["POSTGRES_HOST"], os.environ.get("POSTGRES_PORT","5432"), os.environ["POSTGRES_DB"])
+    pool = await asyncpg.create_pool(dsn=dsn, min_size=1, max_size=2)
+    r = {}
+    now = datetime.now(tz=timezone.utc)
+    cutoff = now - timedelta(hours=24)
+    row = await pool.fetchrow("SELECT avg(extract(epoch from (updated_at - created_at))) as avg_sec, max(extract(epoch from (updated_at - created_at))) as max_sec FROM documents WHERE status = $1 AND updated_at > $2 AND updated_at > created_at", "extracted", cutoff)
+    if row and row["avg_sec"]:
+        r["avg_extraction_sec"] = round(float(row["avg_sec"]), 1)
+        r["max_extraction_sec"] = round(float(row["max_sec"]), 1)
+    latest_trend = await pool.fetchval("SELECT max(generated_at) FROM trend_windows")
+    if latest_trend:
+        r["trend_age_min"] = round((now - latest_trend).total_seconds() / 60)
+    latest_rec = await pool.fetchval("SELECT max(created_at) FROM recommendations")
+    if latest_rec:
+        r["rec_age_min"] = round((now - latest_rec).total_seconds() / 60)
+    await pool.close()
+    print(json.dumps(r))
+asyncio.run(check())
+' 2>/dev/null) || LATENCY_CHECK='{}'
+
+if [[ -n "$LATENCY_CHECK" && "$LATENCY_CHECK" != "{}" ]]; then
+  AVG_EXT=$(echo "$LATENCY_CHECK" | python3 -c "import json,sys; v=json.load(sys.stdin).get('avg_extraction_sec'); print(f'{v}s' if v else 'N/A')")
+  MAX_EXT=$(echo "$LATENCY_CHECK" | python3 -c "import json,sys; v=json.load(sys.stdin).get('max_extraction_sec'); print(f'{v}s' if v else 'N/A')")
+  TREND_AGE=$(echo "$LATENCY_CHECK" | python3 -c "import json,sys; v=json.load(sys.stdin).get('trend_age_min'); print(f'{v}min' if v else 'N/A')")
+  REC_AGE=$(echo "$LATENCY_CHECK" | python3 -c "import json,sys; v=json.load(sys.stdin).get('rec_age_min'); print(f'{v}min' if v else 'N/A')")
+
+  pass "Doc ingestion→extraction: avg=${AVG_EXT}, max=${MAX_EXT}"
+  if echo "$LATENCY_CHECK" | python3 -c "import json,sys; v=json.load(sys.stdin).get('trend_age_min',999); exit(0 if v < 30 else 1)" 2>/dev/null; then
+    pass "Latest trend: ${TREND_AGE} ago"
+  else
+    warn "Latest trend: ${TREND_AGE} ago"
+  fi
+  if echo "$LATENCY_CHECK" | python3 -c "import json,sys; v=json.load(sys.stdin).get('rec_age_min',999); exit(0 if v < 30 else 1)" 2>/dev/null; then
+    pass "Latest recommendation: ${REC_AGE} ago"
+  else
+    warn "Latest recommendation: ${REC_AGE} ago"
+  fi
+else
+  warn "Could not measure pipeline latency"
+fi
+
+# ─── SUMMARY ──────────────────────────────────────────────────
+header "Summary"
+echo -e "  ${GREEN}PASS: $PASS${NC}  ${YELLOW}WARN: $WARN${NC}  ${RED}FAIL: $FAIL${NC}"
+echo ""
+
+if [[ "$FAIL" -gt 0 ]]; then
+  echo -e "${RED}Pipeline has failures that need attention.${NC}"
+  exit 1
+elif [[ "$WARN" -gt 3 ]]; then
+  echo -e "${YELLOW}Pipeline is running but has multiple warnings.${NC}"
+  exit 0
+else
+  echo -e "${GREEN}Pipeline is healthy and flowing.${NC}"
+  exit 0
+fi

services/adapters/broker_service.py

@@ -105,12 +105,12 @@ INSERT INTO orders (
     id, recommendation_id, broker_account_id, ticker, side, order_type,
     quantity, limit_price, stop_price, status, idempotency_key,
     broker_order_id, decision_trace, submitted_at, filled_at,
-    fill_price, fill_quantity
+    fill_price, fill_quantity, rejection_reason, rejected_at
 ) VALUES (
     $1::uuid, $2, $3::uuid, $4, $5, $6,
     $7, $8, $9, $10, $11,
     $12, $13::jsonb, $14, $15,
-    $16, $17
+    $16, $17, $18, $19
 )
 ON CONFLICT (idempotency_key) DO UPDATE SET
     status = EXCLUDED.status,
@@ -118,6 +118,8 @@ ON CONFLICT (idempotency_key) DO UPDATE SET
     filled_at = EXCLUDED.filled_at,
     fill_price = EXCLUDED.fill_price,
     fill_quantity = EXCLUDED.fill_quantity,
+    rejection_reason = COALESCE(EXCLUDED.rejection_reason, orders.rejection_reason),
+    rejected_at = COALESCE(EXCLUDED.rejected_at, orders.rejected_at),
     updated_at = NOW()
 """
 
@@ -361,6 +363,8 @@ async def persist_order(
     """Persist order, events, and risk evaluation to PostgreSQL."""
     now = datetime.now(timezone.utc)
     filled_at = now if resp.status == OrderStatus.FILLED else None
+    rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None
+    rejected_at = now if resp.status == OrderStatus.REJECTED else None
 
     decision_trace = {
         "risk_evaluation": risk_eval,
@@ -389,6 +393,8 @@ async def persist_order(
                 filled_at,
                 resp.filled_avg_price,
                 resp.filled_quantity,
+                rejection_reason,
+                rejected_at,
             )
 
             # Record order events

services/aggregation/bayesian.py

@@ -5,14 +5,20 @@ log-likelihood accumulation, Beta distribution parameters, and
 Shannon entropy for mixed-signal detection.
 
 Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 9.1, 9.7
+V3 posterior assembly: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7
 """
 from __future__ import annotations
 
 import math
 from dataclasses import dataclass
+from typing import TYPE_CHECKING
 
 from services.aggregation.scoring import WeightedSignal
 
+if TYPE_CHECKING:
+    from services.aggregation.regime import V3RegimeClassification
+    from services.aggregation.worker import EvidenceCluster
+
 
 @dataclass(frozen=True)
 class BayesianPosterior:
@@ -125,3 +131,126 @@ def compute_bayesian_posterior(
         entropy=entropy,
         signal_count=count,
     )
+
+
+# ---------------------------------------------------------------------------
+# V3 Posterior Assembly — Calibrated Evidence Engine
+# Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7
+# ---------------------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class V3Posterior:
+    """V3 posterior result from log-odds Bayesian assembly.
+
+    Attributes:
+        p_up: Posterior probability of upward move, (0, 1).
+        p_down: 1 - p_up.
+        log_odds: Raw log-odds (logit) of P_up.
+        strength: abs(2 × P_up - 1), signal conviction [0, 1].
+        direction: Classified direction string ('bullish', 'bearish', 'neutral').
+        n_eff_total: Total effective evidence count across all clusters.
+        regime: Market regime string used for this computation.
+    """
+
+    p_up: float
+    p_down: float
+    log_odds: float
+    strength: float
+    direction: str
+    n_eff_total: float
+    regime: str
+
+
+# Regime-specific direction thresholds: (bullish_threshold, bearish_threshold)
+# P_up >= bullish → "bullish"; P_up <= bearish → "bearish"; else "neutral"
+_V3_DIRECTION_THRESHOLDS: dict[str, tuple[float, float]] = {
+    "panic": (0.68, 0.32),
+    "trend_following": (0.60, 0.40),
+    "mean_reversion": (0.63, 0.37),
+    "uncertainty": (0.65, 0.35),
+}
+
+
+def _logit(p: float) -> float:
+    """Compute logit = ln(p / (1-p)) with boundary guard."""
+    p = max(1e-10, min(1 - 1e-10, p))
+    return math.log(p / (1 - p))
+
+
+def _sigmoid(x: float) -> float:
+    """Compute sigmoid = 1 / (1 + exp(-x)) with overflow guard."""
+    if x > 500:
+        return 1.0
+    if x < -500:
+        return 0.0
+    return 1.0 / (1.0 + math.exp(-x))
+
+
+def compute_v3_posterior(
+    clusters: list[EvidenceCluster],
+    regime: V3RegimeClassification,
+    p_prior: float = 0.50,
+) -> V3Posterior:
+    """Assemble v3 posterior via log-odds accumulation.
+
+    Computes:
+        logit(P_up) = logit(P_prior) + sum(gamma_regime × LLR_c)
+        P_up = sigmoid(log_odds), clamped to [1e-10, 1 - 1e-10]
+        strength = abs(2 × P_up - 1)
+        direction via regime-specific thresholds
+
+    Args:
+        clusters: List of EvidenceCluster objects with computed cluster_llr.
+        regime: V3RegimeClassification providing evidence_multiplier and regime.
+        p_prior: Calibrated prior probability, clamped to [0.40, 0.60].
+
+    Returns:
+        V3Posterior with computed posterior fields.
+
+    Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7
+    """
+    # Clamp prior to [0.40, 0.60] (Req 5.6)
+    p_prior = max(0.40, min(0.60, p_prior))
+
+    # Gamma: regime-specific evidence multiplier (Req 5.2)
+    gamma = regime.evidence_multiplier
+
+    # Compute log-odds: logit(P_prior) + sum(gamma × LLR_c) (Req 5.1, 5.2)
+    log_odds = _logit(p_prior) + sum(gamma * c.cluster_llr for c in clusters)
+
+    # Compute P_up via sigmoid (Req 5.3)
+    p_up = _sigmoid(log_odds)
+    # Clamp to open interval (Req 5.3)
+    p_up = max(1e-10, min(1 - 1e-10, p_up))
+
+    p_down = 1.0 - p_up
+
+    # Strength = |2 × P_up - 1| (Req 5.4)
+    strength = abs(2.0 * p_up - 1.0)
+
+    # n_eff_total = sum of cluster n_eff (Req 5.5)
+    n_eff_total = sum(c.n_eff for c in clusters)
+
+    # Classify direction using regime-specific thresholds (Req 5.5)
+    regime_key = regime.regime.value  # MarketRegime enum → string
+    bull_thresh, bear_thresh = _V3_DIRECTION_THRESHOLDS.get(
+        regime_key, (0.65, 0.35)
+    )
+
+    if p_up >= bull_thresh:
+        direction = "bullish"
+    elif p_up <= bear_thresh:
+        direction = "bearish"
+    else:
+        direction = "neutral"
+
+    return V3Posterior(
+        p_up=p_up,
+        p_down=p_down,
+        log_odds=log_odds,
+        strength=strength,
+        direction=direction,
+        n_eff_total=n_eff_total,
+        regime=regime_key,
+    )

services/aggregation/contradiction.py

@@ -10,10 +10,14 @@ from __future__ import annotations
 
 import math
 from dataclasses import dataclass
+from typing import TYPE_CHECKING
 
 from services.aggregation.scoring import WeightedSignal
 from services.shared.schemas import DisagreementDetail
 
+if TYPE_CHECKING:
+    from services.aggregation.worker import EvidenceCluster
+
 
 @dataclass
 class CatalystEntry:
@@ -236,3 +240,67 @@ def _detect_catalyst_disagreement(
         ))
 
     return details
+
+
+# ---------------------------------------------------------------------------
+# V3 LLR Entropy Contradiction
+# ---------------------------------------------------------------------------
+
+
+def compute_v3_contradiction(clusters: list[EvidenceCluster]) -> float:
+    """Compute LLR entropy contradiction score.
+
+    Uses Shannon entropy over positive/negative cluster LLR magnitudes,
+    weighted by a volume factor that grows with total evidence mass.
+
+    Formula:
+        E_pos = sum(max(LLR_c, 0))
+        E_neg = sum(max(-LLR_c, 0))
+        E_total = E_pos + E_neg
+        f_pos = E_pos / E_total, f_neg = E_neg / E_total
+        H_conflict = -f_pos × log2(f_pos) - f_neg × log2(f_neg)
+        volume_factor = 1 - exp(-E_total / 3.0)
+        result = H_conflict × volume_factor, bounded in [0.0, 1.0]
+
+    Returns 0.0 when:
+        - clusters is empty
+        - E_total == 0 (all cluster LLRs are zero)
+        - Only one direction exists (E_pos == 0 or E_neg == 0)
+
+    Requirements: 7.1–7.7
+    """
+    if not clusters:
+        return 0.0
+
+    e_pos = 0.0
+    e_neg = 0.0
+    for cluster in clusters:
+        llr_c = cluster.cluster_llr
+        if llr_c > 0.0:
+            e_pos += llr_c
+        elif llr_c < 0.0:
+            e_neg += -llr_c  # max(-LLR_c, 0) when LLR_c < 0
+
+    e_total = e_pos + e_neg
+
+    # No evidence or unidirectional → no contradiction
+    if e_total == 0.0:
+        return 0.0
+    if e_pos == 0.0 or e_neg == 0.0:
+        return 0.0
+
+    # Compute fractions
+    f_pos = e_pos / e_total
+    f_neg = e_neg / e_total
+
+    # Shannon entropy H_conflict = -f_pos × log2(f_pos) - f_neg × log2(f_neg)
+    # 0 × log2(0) is treated as 0, but the early returns above guarantee
+    # both f_pos and f_neg are positive here.
+    h_conflict = -f_pos * math.log2(f_pos) - f_neg * math.log2(f_neg)
+
+    # Volume factor: suppresses score when total evidence mass is small
+    volume_factor = 1.0 - math.exp(-e_total / 3.0)
+
+    # Final score bounded to [0.0, 1.0]
+    result = h_conflict * volume_factor
+    return max(0.0, min(1.0, result))

services/aggregation/interpolation.py

@@ -11,6 +11,7 @@ from __future__ import annotations
 
 import json
 import logging
+import math
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
 
@@ -955,3 +956,107 @@ def apply_accelerated_decay(
         return accelerated
 
     return standard_decay
+
+# ---------------------------------------------------------------------------
+# V3 Macro Layer — Noisy-OR Exposure & LLR Emission (Requirements: 9.1–9.5)
+# ---------------------------------------------------------------------------
+
+
+# Noisy-OR weights per dimension
+_V3_MACRO_WEIGHTS: dict[str, float] = {
+    "geo": 0.35,
+    "supply": 0.25,
+    "commodity": 0.25,
+    "sector": 0.15,
+}
+
+# Resilience dampener per tier
+_V3_RESILIENCE_DAMPENER: dict[str, float] = {
+    "global_leader": 0.70,
+    "multinational": 0.85,
+    "regional": 1.00,
+    "domestic": 1.20,
+}
+
+
+def compute_normalized_macro_exposure(
+    overlaps: dict[str, float],
+    tier: str = "regional",
+) -> float:
+    """Compute normalized macro exposure via noisy-OR.
+
+    E_raw = 1 - product(1 - w_k × O_k)
+    E_max = 1 - product(1 - w_k)
+    E_macro = E_raw / E_max × resilience_dampener
+
+    Args:
+        overlaps: Dimension overlap values keyed by 'geo', 'supply',
+                  'commodity', 'sector'. Missing keys treated as 0.0.
+        tier: Market position tier for resilience dampening.
+
+    Returns:
+        Normalized macro exposure in [0, ∞) (can exceed 1.0 for domestic
+        tier due to 1.20 dampener, but typically in [0, ~1.2]).
+
+    Requirements: 9.1, 9.2, 9.3
+    """
+    # E_raw = 1 - product(1 - w_k × O_k)
+    product_raw = 1.0
+    for dim, weight in _V3_MACRO_WEIGHTS.items():
+        o_k = max(0.0, min(1.0, overlaps.get(dim, 0.0)))
+        product_raw *= (1.0 - weight * o_k)
+    e_raw = 1.0 - product_raw
+
+    # E_max = 1 - product(1 - w_k) — theoretical max when all overlaps = 1.0
+    product_max = 1.0
+    for weight in _V3_MACRO_WEIGHTS.values():
+        product_max *= (1.0 - weight)
+    e_max = 1.0 - product_max
+
+    # Guard against zero (should never happen with default weights)
+    if e_max <= 0.0:
+        return 0.0
+
+    # Normalize to [0, 1]
+    e_macro = e_raw / e_max
+
+    # Apply resilience dampener per tier
+    dampener = _V3_RESILIENCE_DAMPENER.get(tier, 1.0)
+    return e_macro * dampener
+
+
+def compute_macro_llr(
+    macro_impact: float,
+    event_confidence: float,
+    q_recency: float,
+    macro_direction: int,
+) -> float:
+    """Compute macro LLR for shared posterior.
+
+    p_macro = clamp(0.50 + 0.30 × macro_impact × event_confidence × q_recency, 0.501, 0.80)
+    LLR_macro = macro_direction × ln(p_macro / (1 - p_macro))
+
+    When macro_direction == 0, returns 0.0 (neutral — no directional signal).
+
+    Args:
+        macro_impact: Normalized macro impact score (typically [0, 1]).
+        event_confidence: Event classification confidence [0, 1].
+        q_recency: Recency quality factor [0, 1].
+        macro_direction: +1 for positive, -1 for negative, 0 for neutral.
+
+    Returns:
+        Log-likelihood ratio for the macro signal. Feeds directly into the
+        shared posterior without separate post-hoc modifier.
+
+    Requirements: 9.4, 9.5
+    """
+    if macro_direction == 0:
+        return 0.0
+
+    # p_macro = clamp(0.50 + 0.30 × macro_impact × event_confidence × q_recency, 0.501, 0.80)
+    p_macro = 0.50 + 0.30 * macro_impact * event_confidence * q_recency
+    p_macro = max(0.501, min(0.80, p_macro))
+
+    # LLR_macro = macro_direction × ln(p_macro / (1 - p_macro))
+    llr = macro_direction * math.log(p_macro / (1.0 - p_macro))
+    return llr

services/aggregation/projection.py

@@ -13,11 +13,15 @@ import logging
 import math
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
+from typing import TYPE_CHECKING
 
 import asyncpg
 
 from services.shared.schemas import TrendSummary
 
+if TYPE_CHECKING:
+    from services.aggregation.regime import V3RegimeClassification
+
 logger = logging.getLogger("projection")
 
 # ---------------------------------------------------------------------------
@@ -493,3 +497,101 @@ async def persist_trend_projection(
         projection.diverges_from_current,
     )
     return str(row_id)
+
+
+# ---------------------------------------------------------------------------
+# V3 Posterior State Projection (Requirements: 11.1–11.7)
+# ---------------------------------------------------------------------------
+
+
+# Regime decay factors (phi) — Req 11.3
+_V3_PHI_DECAY: dict[str, float] = {
+    "panic": 0.35,
+    "trend_following": 0.80,
+    "mean_reversion": 0.55,
+    "uncertainty": 0.50,
+}
+
+
+def _logit(p: float) -> float:
+    """Compute logit = ln(p / (1-p)) with boundary guard."""
+    p = max(1e-10, min(1 - 1e-10, p))
+    return math.log(p / (1 - p))
+
+
+def _sigmoid(x: float) -> float:
+    """Compute sigmoid = 1 / (1 + exp(-x)) with overflow guard."""
+    if x > 500:
+        return 1.0
+    if x < -500:
+        return 0.0
+    return 1.0 / (1.0 + math.exp(-x))
+
+
+@dataclass
+class V3ProjectionState:
+    """V3 posterior state projection result.
+
+    Attributes:
+        a_t: Accumulated evidence state A_t.
+        p_up_projected: Projected probability sigmoid(logit(P_prior) + phi^h * A_t).
+        projected_strength: abs(2 * P_up_projected - 1).
+        diverges: True when sign(P_up_projected - 0.5) != sign(P_up_t - 0.5).
+        phi_regime: Regime-specific decay factor used.
+    """
+
+    a_t: float
+    p_up_projected: float
+    projected_strength: float
+    diverges: bool
+    phi_regime: float
+
+
+def compute_v3_projection(
+    a_prev: float,
+    cluster_llrs: list[float],
+    regime: V3RegimeClassification,
+    p_prior: float,
+    projection_horizon: int,
+    known_catalyst_llr: float = 0.0,
+) -> V3ProjectionState:
+    """Compute posterior state projection with regime-aware decay.
+
+    Evidence state: A_t = phi_regime * A_{t-1} + sum(LLR_c), init A_0 = 0.0
+    Projected alpha: A_projected = phi^h * A_t + known_catalyst_LLR
+    P_up_projected = sigmoid(logit(P_prior) + A_projected)
+    Projected strength = abs(2 * P_up_projected - 1)
+    Divergence flagged when sign(P_up_projected - 0.5) != sign(P_up_t - 0.5)
+
+    Requirements: 11.1–11.7
+    """
+    # Resolve phi from regime; default to uncertainty (0.50) if unavailable (Req 11.7)
+    phi = _V3_PHI_DECAY.get(regime.regime.value, 0.50) if regime else 0.50
+
+    # Evidence state update: A_t = phi * A_{t-1} + sum(LLR_c) — Req 11.1, 11.2
+    a_t = phi * a_prev + sum(cluster_llrs)
+
+    # Projected alpha: A_projected = phi^h * A_t + known_catalyst_LLR — Req 11.4
+    a_projected = (phi ** projection_horizon) * a_t + known_catalyst_llr
+
+    # P_up_projected = sigmoid(logit(P_prior) + A_projected) — Req 11.5
+    p_up_projected = _sigmoid(_logit(p_prior) + a_projected)
+
+    # Projected strength = abs(2 * P_up_projected - 1) — Req 11.6
+    projected_strength = abs(2.0 * p_up_projected - 1.0)
+
+    # Compute current P_up_t for divergence check (not projected)
+    p_up_t = _sigmoid(_logit(p_prior) + a_t)
+
+    # Flag divergence when projected direction differs from current — Req 11.6
+    sign_projected = (p_up_projected - 0.5) >= 0
+    sign_current = (p_up_t - 0.5) >= 0
+    diverges = sign_projected != sign_current
+
+    return V3ProjectionState(
+        a_t=a_t,
+        p_up_projected=p_up_projected,
+        projected_strength=projected_strength,
+        diverges=diverges,
+        phi_regime=phi,
+    )

services/aggregation/regime.py

@@ -168,3 +168,151 @@ def classify_regime(
         bearish_threshold=-threshold,
         contradiction_penalty_multiplier=contradiction_mult,
     )
+
+
+# ---------------------------------------------------------------------------
+# V3 Regime Detection — Calibrated Evidence Engine
+# Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8
+# ---------------------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class V3RegimeClassification:
+    """V3 regime classification result with calibrated parameters.
+
+    Attributes:
+        regime: Market regime category.
+        trend_z: ATR-normalized trend indicator (EMA_20 - EMA_100) / ATR_20.
+        vol_ratio: Volatility ratio sigma_20 / sigma_100.
+        evidence_multiplier: Regime-specific gamma for posterior LLR scaling.
+        confidence_multiplier: Regime-specific confidence scaling factor.
+        phi_decay: Evidence state decay factor for projection.
+        atr_multiplier: Regime-specific ATR multiplier for stop computation.
+    """
+
+    regime: MarketRegime
+    trend_z: float
+    vol_ratio: float
+    evidence_multiplier: float
+    confidence_multiplier: float
+    phi_decay: float
+    atr_multiplier: float
+
+
+# Regime parameter lookup: (gamma, confidence_mult, phi, ATR_mult, min_edge)
+_V3_REGIME_PARAMS: dict[MarketRegime, tuple[float, float, float, float, float]] = {
+    MarketRegime.PANIC: (0.70, 0.70, 0.35, 2.5, 0.0100),
+    MarketRegime.TREND_FOLLOWING: (1.10, 1.00, 0.80, 1.8, 0.0035),
+    MarketRegime.MEAN_REVERSION: (0.90, 0.95, 0.55, 1.4, 0.0050),
+    MarketRegime.UNCERTAINTY: (0.80, 0.85, 0.50, 2.0, 0.0075),
+}
+
+# Default uncertainty classification for v3 when data is insufficient (Req 6.8)
+_DEFAULT_V3_UNCERTAINTY = V3RegimeClassification(
+    regime=MarketRegime.UNCERTAINTY,
+    trend_z=0.0,
+    vol_ratio=1.0,
+    evidence_multiplier=0.80,
+    confidence_multiplier=0.85,
+    phi_decay=0.50,
+    atr_multiplier=2.0,
+)
+
+
+def _compute_ema_full(values: list[float], span: int) -> float:
+    """Compute EMA over the full values list with given span.
+
+    Uses standard EMA formula: alpha = 2 / (span + 1), iterating from the
+    beginning of the list. Seeds EMA with the first value.
+
+    This differs from ``compute_ema`` which only uses the last ``period``
+    values. V3 requires iterating over the full history to produce a stable
+    EMA_100.
+    """
+    if not values or span < 1:
+        raise ValueError("values must be non-empty and span must be >= 1")
+
+    alpha = 2.0 / (span + 1)
+    ema = values[0]
+    for value in values[1:]:
+        ema = alpha * value + (1.0 - alpha) * ema
+    return ema
+
+
+def classify_regime_v3(
+    closing_prices: list[float],
+    daily_returns: list[float],
+    atr_20: float,
+) -> V3RegimeClassification:
+    """Classify market regime using v3 ATR-normalized indicators.
+
+    Computes trend_z = (EMA_20 - EMA_100) / ATR_20 and
+    vol_ratio = sigma_20 / sigma_100 to determine the market regime.
+
+    Classification priority (Req 6.2–6.5):
+        1. Panic: vol_ratio > 1.5 OR |trend_z| > 2.5
+        2. Trend following: |trend_z| >= 0.75 AND vol_ratio < 1.3
+        3. Mean reversion: |trend_z| < 0.50 AND vol_ratio < 1.0
+        4. Uncertainty: all other cases
+
+    Falls back to uncertainty when data is insufficient (Req 6.8):
+        - Fewer than 100 closing prices for EMA_100
+        - ATR_20 <= 0 (insufficient bars for ATR)
+        - Fewer than 100 daily returns for sigma_100
+
+    Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8
+    """
+    # --- Data sufficiency check (Req 6.8) ---
+    if len(closing_prices) < 100:
+        return _DEFAULT_V3_UNCERTAINTY
+
+    if atr_20 <= 0.0:
+        return _DEFAULT_V3_UNCERTAINTY
+
+    if len(daily_returns) < 100:
+        return _DEFAULT_V3_UNCERTAINTY
+
+    # --- Compute trend_z (Req 6.1) ---
+    ema_20 = _compute_ema_full(closing_prices, span=20)
+    ema_100 = _compute_ema_full(closing_prices, span=100)
+    trend_z = (ema_20 - ema_100) / atr_20
+
+    # --- Compute vol_ratio (Req 6.1) ---
+    sigma_20 = statistics.stdev(daily_returns[-20:]) if len(daily_returns) >= 20 else 0.0
+    sigma_100 = statistics.stdev(daily_returns[-100:])
+
+    # Guard against zero sigma_100
+    if sigma_100 <= 0.0 or math.isnan(sigma_100):
+        return _DEFAULT_V3_UNCERTAINTY
+
+    if math.isnan(sigma_20):
+        return _DEFAULT_V3_UNCERTAINTY
+
+    vol_ratio = sigma_20 / sigma_100
+
+    # --- Classification rules (Req 6.2–6.5) ---
+    # Priority 1: Panic (Req 6.2)
+    if vol_ratio > 1.5 or abs(trend_z) > 2.5:
+        regime = MarketRegime.PANIC
+    # Priority 2: Trend following (Req 6.3)
+    elif abs(trend_z) >= 0.75 and vol_ratio < 1.3:
+        regime = MarketRegime.TREND_FOLLOWING
+    # Priority 3: Mean reversion (Req 6.4)
+    elif abs(trend_z) < 0.50 and vol_ratio < 1.0:
+        regime = MarketRegime.MEAN_REVERSION
+    # Priority 4: Uncertainty (Req 6.5)
+    else:
+        regime = MarketRegime.UNCERTAINTY
+
+    # --- Assign regime parameters (Req 6.6, 6.7) ---
+    gamma, conf_mult, phi, atr_mult, _min_edge = _V3_REGIME_PARAMS[regime]
+
+    return V3RegimeClassification(
+        regime=regime,
+        trend_z=trend_z,
+        vol_ratio=vol_ratio,
+        evidence_multiplier=gamma,
+        confidence_multiplier=conf_mult,
+        phi_decay=phi,
+        atr_multiplier=atr_mult,
+    )

services/aggregation/scoring.py

@@ -8,9 +8,14 @@ Requirements: 2.1–2.6, 3.1–3.5, 4.2–4.3, 5.1–5.7, 6.1–6.5, 16.4–16.5
 """
 from __future__ import annotations
 
+import hashlib
+import logging
 import math
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
+from typing import Any
+
+import asyncpg
 
 from services.shared.schemas import MarketContext
 
@@ -574,6 +579,57 @@ def sentiment_to_numeric(sentiment: str) -> float:
     return mapping.get(sentiment.lower(), 0.0)
 
 
+async def normalize_impact_scores(
+    pool: asyncpg.Pool,
+    ticker: str,
+    raw_scores: list[float],
+) -> list[float]:
+    """Normalize impact scores using 7-day rolling z-score per ticker.
+
+    Computes the 7-day mean and stddev of impact_score values from
+    document_impact_records for the given ticker, then normalizes
+    each raw score: normalized = (raw - mean_7d) / max(stddev_7d, 0.1)
+
+    The 0.1 floor prevents division by near-zero stddev for low-activity tickers.
+
+    Fallback: if fewer than 5 records in the 7-day window, returns raw scores
+    unchanged (insufficient data for meaningful normalization).
+
+    Args:
+        pool: asyncpg connection pool.
+        ticker: The ticker symbol to query distribution for.
+        raw_scores: List of raw impact_score values to normalize.
+
+    Returns:
+        List of normalized impact scores (same length as input).
+    """
+    if not raw_scores:
+        return []
+
+    row = await pool.fetchrow(
+        """
+        SELECT AVG(impact_score) as mean,
+               STDDEV(impact_score) as stddev,
+               COUNT(*) as cnt
+        FROM document_impact_records
+        WHERE ticker = $1 AND created_at >= NOW() - INTERVAL '7 days'
+        """,
+        ticker,
+    )
+
+    # Fallback: insufficient data for meaningful normalization
+    if row is None or row["cnt"] < 5:
+        return list(raw_scores)
+
+    mean_7d: float = float(row["mean"])
+    stddev_7d: float = float(row["stddev"]) if row["stddev"] is not None and row["stddev"] > 0 else 0.0
+
+    # Apply 0.1 floor to prevent division by near-zero stddev
+    effective_stddev = max(stddev_7d, 0.1)
+
+    return [(raw - mean_7d) / effective_stddev for raw in raw_scores]
+
+
 def weighted_sentiment_average(signals: list[WeightedSignal]) -> float:
     """Compute a weight-adjusted average sentiment across signals.
 
@@ -588,3 +644,625 @@ def weighted_sentiment_average(signals: list[WeightedSignal]) -> float:
     if total_weight == 0.0:
         return 0.0
     return weighted_sum / total_weight
+
+
+# ===========================================================================
+# V3 Calibrated Evidence Engine — EvidenceUnit and Normalization
+# ===========================================================================
+# All code below this line implements the v3 pipeline. It is gated behind
+# the `v3_engine_enabled` feature flag at the worker/orchestration layer.
+# ===========================================================================
+
+
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# V3 Event type base rates (expanded for v3 pipeline)
+# ---------------------------------------------------------------------------
+
+V3_EVENT_TYPE_BASE_RATES: dict[str, float] = {
+    "earnings": 0.25,
+    "guidance": 0.20,
+    "merger_acquisition": 0.05,
+    "product_launch": 0.15,
+    "regulatory": 0.10,
+    "management_change": 0.08,
+    "partnership": 0.12,
+    "legal": 0.07,
+    "analyst_rating": 0.30,
+    "market_data": 0.40,
+}
+V3_DEFAULT_BASE_RATE: float = 0.10
+
+# Direction mapping constants
+_POSITIVE_DIRECTIONS: frozenset[str] = frozenset({"positive", "bullish"})
+_NEGATIVE_DIRECTIONS: frozenset[str] = frozenset({"negative", "bearish"})
+_NEUTRAL_DIRECTIONS: frozenset[str] = frozenset({"neutral", "mixed"})
+
+# Macro horizon mapping
+_MACRO_HORIZON_MAP: dict[str, str] = {
+    "short_term": "7d",
+    "medium_term": "30d",
+    "long_term": "90d",
+}
+
+# Valid horizons
+_VALID_HORIZONS: frozenset[str] = frozenset({"intraday", "1d", "7d", "30d", "90d"})
+
+
+# ---------------------------------------------------------------------------
+# EvidenceUnit dataclass
+# ---------------------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class EvidenceUnit:
+    """Canonical normalized signal representation for the v3 pipeline.
+
+    Every signal — company, macro, or competitive — is normalized into this
+    shape before entering the calibrated reliability / LLR pipeline.
+    """
+
+    symbol: str
+    layer: str  # "company" | "macro" | "competitive"
+    event_type: str
+    source_id: str
+    source_group: str
+    timestamp: datetime
+    horizon: str  # "intraday" | "1d" | "7d" | "30d" | "90d"
+    direction: int  # -1, 0, +1
+    sentiment_strength: float  # [0, 1]
+    impact: float  # [0, 1]
+    extraction_conf: float  # [0, 1]
+    source_cred: float  # [0, 1]
+    novelty: float  # [0, 1]
+    event_base_rate: float  # (0, 1]
+    cluster_id: str
+
+
+# ---------------------------------------------------------------------------
+# Helper functions
+# ---------------------------------------------------------------------------
+
+
+def _map_direction(direction_str: str | None) -> int:
+    """Map a sentiment/impact_direction string to a numeric direction.
+
+    Returns:
+        +1 for positive/bullish, -1 for negative/bearish, 0 for neutral/mixed/unknown.
+    """
+    if direction_str is None:
+        return 0
+    lowered = direction_str.lower().strip()
+    if lowered in _POSITIVE_DIRECTIONS:
+        return 1
+    if lowered in _NEGATIVE_DIRECTIONS:
+        return -1
+    return 0
+
+
+def _get_event_base_rate(event_type: str | None) -> float:
+    """Look up base rate for an event type, defaulting to 0.10."""
+    if event_type is None:
+        return V3_DEFAULT_BASE_RATE
+    return V3_EVENT_TYPE_BASE_RATES.get(event_type, V3_DEFAULT_BASE_RATE)
+
+
+def _compute_cluster_id(
+    symbol: str,
+    horizon: str,
+    event_type: str,
+    source_group: str,
+    time_bucket: str,
+) -> str:
+    """Compute a deterministic cluster_id from the grouping key."""
+    key = f"{symbol}|{horizon}|{event_type}|{source_group}|{time_bucket}"
+    return hashlib.sha256(key.encode()).hexdigest()[:16]
+
+
+def _default_time_bucket(ts: datetime, horizon: str) -> str:
+    """Compute a time bucket string for clustering based on horizon.
+
+    Bucket resolution per horizon:
+        intraday → 1h, 1d → 4h, 7d → 24h, 30d → 72h, 90d → 168h
+    """
+    bucket_hours: dict[str, int] = {
+        "intraday": 1,
+        "1d": 4,
+        "7d": 24,
+        "30d": 72,
+        "90d": 168,
+    }
+    hours = bucket_hours.get(horizon, 24)
+    # Truncate timestamp to bucket boundary
+    epoch_hours = int(ts.timestamp() / 3600)
+    bucket_start = (epoch_hours // hours) * hours
+    return str(bucket_start)
+
+
+def _safe_float(value: Any, default: float = 0.5) -> float:
+    """Extract a float value, substituting default for missing/None."""
+    if value is None:
+        return default
+    try:
+        return float(value)
+    except (TypeError, ValueError):
+        return default
+
+
+def _clamp(value: float, lo: float, hi: float) -> float:
+    """Clamp a value to [lo, hi]."""
+    return max(lo, min(value, hi))
+
+
+# ---------------------------------------------------------------------------
+# Normalization functions
+# ---------------------------------------------------------------------------
+
+
+def normalize_company_signal(
+    signal: dict[str, Any],
+    *,
+    cluster_id: str | None = None,
+) -> EvidenceUnit | None:
+    """Normalize a company signal into an EvidenceUnit.
+
+    Args:
+        signal: Dict with keys from document_impact_records or similar.
+            Required: symbol, timestamp, source_id
+            Optional: event_type, source_group, horizon, sentiment,
+                      sentiment_strength, impact, extraction_conf,
+                      source_cred, novelty
+        cluster_id: If provided, use this cluster_id. Otherwise compute
+            from the signal's grouping key.
+
+    Returns:
+        EvidenceUnit or None if required fields are missing.
+    """
+    # Validate required fields
+    symbol = signal.get("symbol")
+    timestamp = signal.get("timestamp")
+    source_id = signal.get("source_id")
+
+    if not symbol:
+        logger.warning("v3: Rejecting company signal — missing 'symbol'. source: %s", signal.get("source_id", "unknown"))
+        return None
+    if timestamp is None:
+        logger.warning("v3: Rejecting company signal — missing 'timestamp'. symbol=%s, source_id=%s", symbol, source_id)
+        return None
+    if not source_id:
+        logger.warning("v3: Rejecting company signal — missing 'source_id'. symbol=%s", symbol)
+        return None
+
+    # Ensure timestamp is datetime
+    if isinstance(timestamp, str):
+        timestamp = datetime.fromisoformat(timestamp)
+    if timestamp.tzinfo is None:
+        timestamp = timestamp.replace(tzinfo=timezone.utc)
+
+    # Extract and default fields
+    event_type = signal.get("event_type") or "unknown"
+    source_group = signal.get("source_group") or "company"
+    horizon = signal.get("horizon") or "7d"
+    if horizon not in _VALID_HORIZONS:
+        horizon = "7d"
+
+    # Direction mapping
+    direction = _map_direction(signal.get("sentiment") or signal.get("direction"))
+
+    # Optional numeric fields — default to 0.5 if missing
+    sentiment_strength = _clamp(_safe_float(signal.get("sentiment_strength")), 0.0, 1.0)
+    impact = _clamp(_safe_float(signal.get("impact")), 0.0, 1.0)
+    extraction_conf = _clamp(_safe_float(signal.get("extraction_conf") or signal.get("extraction_confidence")), 0.0, 1.0)
+    source_cred = _clamp(_safe_float(signal.get("source_cred") or signal.get("source_credibility")), 0.0, 1.0)
+    novelty = _clamp(_safe_float(signal.get("novelty") or signal.get("novelty_score")), 0.0, 1.0)
+
+    # Event base rate
+    event_base_rate = _get_event_base_rate(event_type)
+
+    # Cluster ID
+    if cluster_id is None:
+        time_bucket = _default_time_bucket(timestamp, horizon)
+        cluster_id = _compute_cluster_id(symbol, horizon, event_type, source_group, time_bucket)
+
+    return EvidenceUnit(
+        symbol=str(symbol),
+        layer="company",
+        event_type=event_type,
+        source_id=str(source_id),
+        source_group=source_group,
+        timestamp=timestamp,
+        horizon=horizon,
+        direction=direction,
+        sentiment_strength=sentiment_strength,
+        impact=impact,
+        extraction_conf=extraction_conf,
+        source_cred=source_cred,
+        novelty=novelty,
+        event_base_rate=event_base_rate,
+        cluster_id=cluster_id,
+    )
+
+
+def normalize_macro_signal(
+    signal: dict[str, Any],
+    *,
+    cluster_id: str | None = None,
+) -> EvidenceUnit | None:
+    """Normalize a macro signal into an EvidenceUnit.
+
+    Macro signals come from macro_impact_records joined with global_events.
+
+    Args:
+        signal: Dict with keys from macro impact/global event records.
+            Required: symbol (or ticker), timestamp, source_id (or event_id)
+            Optional: event_type, impact_direction, macro_impact_score,
+                      event_confidence, estimated_duration, novelty
+        cluster_id: If provided, use this cluster_id.
+
+    Returns:
+        EvidenceUnit or None if required fields are missing.
+    """
+    # Validate required fields
+    symbol = signal.get("symbol") or signal.get("ticker")
+    timestamp = signal.get("timestamp")
+    source_id = signal.get("source_id") or signal.get("event_id")
+
+    if not symbol:
+        logger.warning("v3: Rejecting macro signal — missing 'symbol'/'ticker'. source: %s", signal.get("source_id", "unknown"))
+        return None
+    if timestamp is None:
+        logger.warning("v3: Rejecting macro signal — missing 'timestamp'. symbol=%s, source_id=%s", symbol, source_id)
+        return None
+    if not source_id:
+        logger.warning("v3: Rejecting macro signal — missing 'source_id'/'event_id'. symbol=%s", symbol)
+        return None
+
+    # Ensure timestamp is datetime
+    if isinstance(timestamp, str):
+        timestamp = datetime.fromisoformat(timestamp)
+    if timestamp.tzinfo is None:
+        timestamp = timestamp.replace(tzinfo=timezone.utc)
+
+    # Extract fields
+    event_type = signal.get("event_type") or "unknown"
+    source_group = "macro"
+
+    # Horizon from estimated_duration
+    estimated_duration = signal.get("estimated_duration") or "medium_term"
+    horizon = _MACRO_HORIZON_MAP.get(estimated_duration, "30d")
+
+    # Direction from impact_direction
+    direction = _map_direction(signal.get("impact_direction") or signal.get("direction"))
+
+    # Impact from macro_impact_score
+    impact = _clamp(_safe_float(signal.get("macro_impact_score") or signal.get("impact")), 0.0, 1.0)
+
+    # Source cred and extraction conf from event_confidence
+    event_confidence = _safe_float(signal.get("event_confidence") or signal.get("confidence"))
+    source_cred = _clamp(event_confidence, 0.0, 1.0)
+    extraction_conf = _clamp(event_confidence, 0.0, 1.0)
+
+    # Novelty: 1.0 for new events (as per requirement 1.2)
+    novelty = _clamp(_safe_float(signal.get("novelty"), default=1.0), 0.0, 1.0)
+
+    # Sentiment strength — default 0.5 for macro
+    sentiment_strength = _clamp(_safe_float(signal.get("sentiment_strength")), 0.0, 1.0)
+
+    # Event base rate
+    event_base_rate = _get_event_base_rate(event_type)
+
+    # Cluster ID
+    if cluster_id is None:
+        time_bucket = _default_time_bucket(timestamp, horizon)
+        cluster_id = _compute_cluster_id(str(symbol), horizon, event_type, source_group, time_bucket)
+
+    return EvidenceUnit(
+        symbol=str(symbol),
+        layer="macro",
+        event_type=event_type,
+        source_id=str(source_id),
+        source_group=source_group,
+        timestamp=timestamp,
+        horizon=horizon,
+        direction=direction,
+        sentiment_strength=sentiment_strength,
+        impact=impact,
+        extraction_conf=extraction_conf,
+        source_cred=source_cred,
+        novelty=novelty,
+        event_base_rate=event_base_rate,
+        cluster_id=cluster_id,
+    )
+
+
+def normalize_competitive_signal(
+    signal: dict[str, Any],
+    *,
+    cluster_id: str | None = None,
+) -> EvidenceUnit | None:
+    """Normalize a competitive signal into an EvidenceUnit.
+
+    Competitive signals come from pattern mining and cross-company propagation.
+
+    Args:
+        signal: Dict with keys from competitive_signal_records.
+            Required: symbol (or target_ticker), timestamp, source_id (or source_document_id)
+            Optional: event_type, signal_direction, signal_strength,
+                      relationship_strength, pattern_confidence, time_horizon
+        cluster_id: If provided, use this cluster_id.
+
+    Returns:
+        EvidenceUnit or None if required fields are missing.
+    """
+    # Validate required fields
+    symbol = signal.get("symbol") or signal.get("target_ticker")
+    timestamp = signal.get("timestamp")
+    source_id = signal.get("source_id") or signal.get("source_document_id")
+
+    if not symbol:
+        logger.warning("v3: Rejecting competitive signal — missing 'symbol'/'target_ticker'. source: %s", signal.get("source_id", "unknown"))
+        return None
+    if timestamp is None:
+        logger.warning("v3: Rejecting competitive signal — missing 'timestamp'. symbol=%s, source_id=%s", symbol, source_id)
+        return None
+    if not source_id:
+        logger.warning("v3: Rejecting competitive signal — missing 'source_id'/'source_document_id'. symbol=%s", symbol)
+        return None
+
+    # Ensure timestamp is datetime
+    if isinstance(timestamp, str):
+        timestamp = datetime.fromisoformat(timestamp)
+    if timestamp.tzinfo is None:
+        timestamp = timestamp.replace(tzinfo=timezone.utc)
+
+    # Extract fields
+    event_type = signal.get("event_type") or "unknown"
+    source_group = "competitive"
+
+    # Horizon from time_horizon field
+    time_horizon = signal.get("time_horizon") or signal.get("horizon") or "7d"
+    if time_horizon in _MACRO_HORIZON_MAP:
+        horizon = _MACRO_HORIZON_MAP[time_horizon]
+    elif time_horizon in _VALID_HORIZONS:
+        horizon = time_horizon
+    else:
+        horizon = "7d"
+
+    # Direction from signal_direction (bullish/bearish/neutral)
+    direction = _map_direction(signal.get("signal_direction") or signal.get("direction"))
+
+    # Impact = signal_strength × relationship_strength (Req 1.3)
+    signal_strength = _safe_float(signal.get("signal_strength"))
+    relationship_strength = _safe_float(signal.get("relationship_strength"))
+    impact = _clamp(signal_strength * relationship_strength, 0.0, 1.0)
+
+    # Source cred from pattern_confidence (Req 1.3)
+    pattern_confidence = _safe_float(signal.get("pattern_confidence"))
+    source_cred = _clamp(pattern_confidence, 0.0, 1.0)
+
+    # Extraction conf = pattern_confidence (Req 1.3)
+    extraction_conf = _clamp(pattern_confidence, 0.0, 1.0)
+
+    # Novelty: 1.0 for competitive signals (Req 1.3)
+    novelty = _clamp(_safe_float(signal.get("novelty"), default=1.0), 0.0, 1.0)
+
+    # Sentiment strength — default 0.5 for competitive
+    sentiment_strength = _clamp(_safe_float(signal.get("sentiment_strength")), 0.0, 1.0)
+
+    # Event base rate
+    event_base_rate = _get_event_base_rate(event_type)
+
+    # Cluster ID
+    if cluster_id is None:
+        time_bucket = _default_time_bucket(timestamp, horizon)
+        cluster_id = _compute_cluster_id(str(symbol), horizon, event_type, source_group, time_bucket)
+
+    return EvidenceUnit(
+        symbol=str(symbol),
+        layer="competitive",
+        event_type=event_type,
+        source_id=str(source_id),
+        source_group=source_group,
+        timestamp=timestamp,
+        horizon=horizon,
+        direction=direction,
+        sentiment_strength=sentiment_strength,
+        impact=impact,
+        extraction_conf=extraction_conf,
+        source_cred=source_cred,
+        novelty=novelty,
+        event_base_rate=event_base_rate,
+        cluster_id=cluster_id,
+    )
+
+
+# ===========================================================================
+# V3 Calibrated Reliability Pipeline
+# ===========================================================================
+
+
+@dataclass(frozen=True)
+class SourceStats:
+    """Historical accuracy stats for a signal source (Bayesian prior).
+
+    Used to compute q_source via Beta-Binomial shrinkage.
+    """
+
+    source_id: str
+    hits: int = 0  # correct directional predictions
+    misses: int = 0  # incorrect directional predictions
+    alpha_0: float = 3.0  # Beta prior alpha (pseudo-successes)
+    beta_0: float = 3.0  # Beta prior beta (pseudo-failures)
+
+
+@dataclass(frozen=True)
+class ReliabilityComponents:
+    """Breakdown of calibrated reliability for a single signal.
+
+    Each q_* factor is in [0, 1] and represents one quality dimension.
+    q_i is the final combined reliability used downstream in LLR conversion.
+    """
+
+    q_ext: float  # extraction confidence reliability
+    q_source: float  # source accuracy reliability (Bayesian shrinkage)
+    q_recency: float  # temporal freshness reliability
+    q_uniqueness: float  # novelty / de-duplication reliability
+    q_i: float  # final combined: clamp(q_ext × q_source × source_cred × q_recency × q_uniqueness, 0, 1)
+
+
+# Horizon-specific base half-lives for recency decay (hours)
+_V3_TAU_BASE: dict[str, float] = {
+    "intraday": 2.0,
+    "1d": 12.0,
+    "7d": 72.0,
+    "30d": 240.0,
+    "90d": 720.0,
+}
+
+
+def _sigmoid(x: float) -> float:
+    """Compute sigmoid(x) = 1 / (1 + exp(-x)) with overflow guard."""
+    if x < -500.0:
+        return 0.0
+    if x > 500.0:
+        return 1.0
+    return 1.0 / (1.0 + math.exp(-x))
+
+
+def compute_v3_reliability(
+    unit: EvidenceUnit,
+    source_stats: SourceStats,
+    cluster_position: int,  # duplicate_count_before
+    reference_time: datetime,
+) -> ReliabilityComponents:
+    """Compute calibrated reliability components for an EvidenceUnit.
+
+    Implements Requirements 2.1–2.9: extraction confidence gate, Bayesian
+    source accuracy, adaptive recency decay, and novelty/uniqueness penalty.
+
+    Args:
+        unit: The normalized evidence unit to score.
+        source_stats: Historical accuracy record for the signal's source.
+        cluster_position: Number of signals in the same cluster ingested
+            before this one (duplicate_count_before). 0 for first-in-cluster.
+        reference_time: The "now" anchor for computing age_hours.
+
+    Returns:
+        ReliabilityComponents with individual factors and combined q_i.
+    """
+    # --- q_ext: extraction confidence reliability (Req 2.1) ---
+    # sigmoid(8.0 × (extraction_conf - 0.55))
+    q_ext = _sigmoid(8.0 * (unit.extraction_conf - 0.55))
+
+    # --- q_source: Bayesian shrinkage source reliability (Req 2.2, 2.3) ---
+    alpha = source_stats.alpha_0 + source_stats.hits
+    beta = source_stats.beta_0 + source_stats.misses
+    e_theta = alpha / (alpha + beta)
+    # clamp((E[theta] - 0.50) / 0.35, 0, 1)
+    q_source = _clamp((e_theta - 0.50) / 0.35, 0.0, 1.0)
+
+    # --- q_recency: adaptive exponential decay (Req 2.4, 2.5, 2.6) ---
+    # Ensure tz-aware timestamps
+    ts = unit.timestamp
+    if ts.tzinfo is None:
+        ts = ts.replace(tzinfo=timezone.utc)
+    ref = reference_time
+    if ref.tzinfo is None:
+        ref = ref.replace(tzinfo=timezone.utc)
+
+    age_hours = max((ref - ts).total_seconds() / 3600.0, 0.0)
+
+    # Adaptive half-life: tau_adaptive = tau_base × (1 + 0.75 × impact + 0.50 × surprise)
+    # surprise = clamp(-log2(event_base_rate) / 5, 0, 1)
+    event_base_rate = unit.event_base_rate
+    if event_base_rate <= 0.0:
+        event_base_rate = 0.10  # Req 2.5: default to 0.10 to prevent log(0)
+
+    surprise = _clamp(-math.log2(event_base_rate) / 5.0, 0.0, 1.0)
+
+    tau_base = _V3_TAU_BASE.get(unit.horizon, 72.0)
+    tau_adaptive = tau_base * (1.0 + 0.75 * unit.impact + 0.50 * surprise)
+
+    # q_recency = 2^(-age_hours / tau_adaptive)
+    # Guard against extreme exponents
+    if tau_adaptive <= 0.0:
+        tau_adaptive = tau_base  # fallback
+    exponent = -age_hours / tau_adaptive
+    # For very large negative exponents, result is effectively 0
+    if exponent < -1000.0:
+        q_recency = 0.0
+    else:
+        q_recency = math.pow(2.0, exponent)
+
+    # --- q_uniqueness: novelty + de-duplication (Req 2.7) ---
+    # clamp(0.5 + 0.5 × novelty, 0.5, 1.0) × (1 / sqrt(1 + dup_count))
+    novelty_factor = _clamp(0.5 + 0.5 * unit.novelty, 0.5, 1.0)
+    dedup_factor = 1.0 / math.sqrt(1.0 + cluster_position)
+    q_uniqueness = novelty_factor * dedup_factor
+
+    # --- q_i: combined reliability (Req 2.8) ---
+    q_i = _clamp(
+        q_ext * q_source * unit.source_cred * q_recency * q_uniqueness,
+        0.0,
+        1.0,
+    )
+
+    # --- Explainability floor on q_recency (Req 2.9) ---
+    # Apply floor of 0.01 only for the display value; q_i uses raw q_recency
+    q_recency_display = max(q_recency, 0.01)
+
+    return ReliabilityComponents(
+        q_ext=q_ext,
+        q_source=q_source,
+        q_recency=q_recency_display,
+        q_uniqueness=q_uniqueness,
+        q_i=q_i,
+    )
+
+
+# ---------------------------------------------------------------------------
+# V3 LLR Conversion (Requirements 3.1–3.6)
+# ---------------------------------------------------------------------------
+
+
+def compute_llr(unit: EvidenceUnit, q_i: float) -> float:
+    """Convert calibrated reliability to log-likelihood ratio.
+
+    Requirements: 3.1–3.6
+
+    Formula:
+        p_correct = clamp(0.50 + 0.35 × q_i × impact × sentiment_strength, 0.501, 0.85)
+        LLR_i = direction × ln(p_correct / (1 - p_correct))
+
+    For neutral signals (direction == 0), returns 0.0 immediately.
+    For directional signals, the LLR sign always matches direction.
+
+    Bounds:
+        - Minimum |LLR| ≈ ln(0.501/0.499) ≈ 0.004 for directional signals
+        - Maximum |LLR| ≈ ln(0.85/0.15) ≈ 1.735
+
+    Args:
+        unit: The normalized evidence unit containing direction, impact,
+            and sentiment_strength.
+        q_i: The combined calibrated reliability from compute_v3_reliability.
+
+    Returns:
+        Log-likelihood ratio. Positive for bullish, negative for bearish,
+        zero for neutral.
+    """
+    # Req 3.5: Neutral signals produce zero LLR
+    if unit.direction == 0:
+        return 0.0
+
+    # Req 3.1–3.2: Compute p_correct with calibrated reliability
+    p_correct = _clamp(
+        0.50 + 0.35 * q_i * unit.impact * unit.sentiment_strength,
+        0.501,
+        0.85,
+    )
+
+    # Req 3.3–3.4: LLR_i = direction × ln(p_correct / (1 - p_correct))
+    llr = unit.direction * math.log(p_correct / (1.0 - p_correct))
+
+    return llr

services/aggregation/signal_propagation.py

@@ -378,3 +378,74 @@ def build_pattern_weighted_signals(
         ))
 
     return signals
+
+
+# ---------------------------------------------------------------------------
+# v3 — Correlation-shrunk competitive propagation (Requirements: 10.1–10.5)
+# ---------------------------------------------------------------------------
+
+_V3_MAX_NETWORK_DISTANCE = 3
+
+
+def compute_shrunk_correlation(
+    rho_rolling: float,
+    n_observations: int,
+    same_sector: bool,
+) -> float:
+    """Compute shrinkage-adjusted correlation.
+
+    Shrinks the rolling correlation toward a sector-aware prior using a
+    Bayesian-style weight of n / (n + 30).
+
+    rho_prior = 0.30 if same_sector else 0.10
+    rho_shrunk = (n/(n+30)) × rho_rolling + (30/(n+30)) × rho_prior
+    rho_effective = max(rho_shrunk, 0)
+
+    Args:
+        rho_rolling: Rolling pairwise correlation estimate.
+        n_observations: Number of observations used to compute rho_rolling.
+        same_sector: Whether the two securities are in the same sector.
+
+    Returns:
+        Non-negative shrinkage-adjusted correlation (rho_effective).
+
+    Requirements: 10.1, 10.2
+    """
+    rho_prior = 0.30 if same_sector else 0.10
+    n = n_observations
+    rho_shrunk = (n / (n + 30)) * rho_rolling + (30 / (n + 30)) * rho_prior
+    rho_effective = max(rho_shrunk, 0.0)
+    return rho_effective
+
+
+def compute_competitive_llr(
+    llr_source: float,
+    rho_effective: float,
+    d_network: int,
+    pattern_confidence: float,
+) -> float:
+    """Compute competitive LLR with graph attenuation.
+
+    attenuation = rho_effective × exp(-0.85 × d_network)
+    LLR_competitive = clamp(llr_source × attenuation × pattern_confidence, -1.25, 1.25)
+
+    When d_network > 3 → attenuation = 0 → LLR_competitive = 0
+
+    Args:
+        llr_source: Source signal LLR value.
+        rho_effective: Shrinkage-adjusted correlation (non-negative).
+        d_network: Graph distance between source and target (integer >= 1).
+        pattern_confidence: Confidence of the historical pattern in [0, 1].
+
+    Returns:
+        Competitive LLR clamped to [-1.25, 1.25]. Returns 0.0 when
+        d_network exceeds max distance of 3.
+
+    Requirements: 10.3, 10.4, 10.5
+    """
+    if d_network > _V3_MAX_NETWORK_DISTANCE:
+        return 0.0
+
+    attenuation = rho_effective * math.exp(-0.85 * d_network)
+    llr_competitive = llr_source * attenuation * pattern_confidence
+    return max(-1.25, min(1.25, llr_competitive))

services/api/app.py

@@ -456,7 +456,7 @@ async def list_trend_history(
                        dominant_catalysts, material_risks, generated_at
                 FROM trend_history
                 {where}
-                ORDER BY generated_at ASC
+                ORDER BY generated_at DESC
                 LIMIT ${idx}""",
             *params, limit,
         )
@@ -470,6 +470,9 @@ async def list_trend_history(
         d["dominant_catalysts"] = _parse_jsonb(d.get("dominant_catalysts"))
         d["material_risks"] = _parse_jsonb(d.get("material_risks"))
         results.append(d)
+    # Return in ascending order for chart rendering (query fetches newest first
+    # so the LIMIT captures recent data relevant to short time windows).
+    results.reverse()
     return results
 
 
@@ -496,7 +499,7 @@ async def get_market_prices(
                (data->>'t')::bigint AS bar_timestamp
            FROM market_snapshots
            WHERE ticker = $1 AND snapshot_type = 'bar'
-           ORDER BY captured_at ASC
+           ORDER BY captured_at DESC
            LIMIT $2""",
         ticker, limit,
     )
@@ -521,6 +524,9 @@ async def get_market_prices(
             "bar_timestamp": bar_ts,
             "captured_at": r["captured_at"].isoformat() if r["captured_at"] else None,
         })
+    # Reverse to ascending order for chart rendering (query fetches newest first
+    # so the LIMIT captures recent data relevant to short time windows).
+    results.reverse()
 
     # Compute 90-day high/low from all bars in the window
     cutoff_90d = datetime.now(timezone.utc) - timedelta(days=90)

services/extractor/inference_adapter.py

@@ -0,0 +1,225 @@
+"""Inference adapter bridging the document extractor to the InferenceGateway.
+
+Replaces direct use of llm_factory / VLLMClient / OllamaClient in the
+extraction pipeline. Uses the shared InferenceGateway with extraction-
+specific prompt construction and records actual endpoint, deployment,
+model, and protocol lineage in the result.
+
+Requirements: 2.12, 13.6
+"""
+from __future__ import annotations
+
+import logging
+import time
+from dataclasses import dataclass, field
+
+from services.extractor.client import (
+    ExtractionAttempt,
+    ExtractionResponse,
+    _repair_json,
+    _strip_markdown_fences,
+)
+from services.extractor.prompts import (
+    build_extraction_prompt,
+    get_json_schema,
+    get_prompt_metadata,
+)
+from services.extractor.schemas import validate_extraction
+from services.shared.inference.gateway import InferenceGateway
+from services.shared.inference.lineage import ModelLineage, build_lineage_from_result
+from services.shared.inference.models import (
+    ChatMessage,
+    InferenceResult,
+    InferenceTarget,
+    StructuredGenerationRequest,
+)
+
+logger = logging.getLogger("extractor.inference_adapter")
+
+
+@dataclass
+class ExtractionWithLineage:
+    """Extraction result bundled with inference lineage metadata.
+
+    The lineage records the actual endpoint, deployment, model, and
+    protocol used — fixing the hardcoded ``model_provider = 'ollama'``.
+    """
+
+    response: ExtractionResponse
+    lineage: ModelLineage
+    raw_inference_results: list[InferenceResult] = field(default_factory=list)
+
+
+async def extract_document(
+    gateway: InferenceGateway,
+    target: InferenceTarget,
+    document_text: str,
+    document_type: str = "article",
+    document_id: str = "",
+    known_tickers: list[str] | None = None,
+    max_retries: int = 3,
+    retry_base_delay: float = 2.0,
+    retry_max_delay: float = 30.0,
+    retry_backoff_multiplier: float = 2.0,
+) -> ExtractionWithLineage:
+    """Extract structured intelligence from a document via the InferenceGateway.
+
+    This adapter:
+    1. Builds extraction-specific prompts (same as current pipeline)
+    2. Constructs a StructuredGenerationRequest
+    3. Routes through the InferenceGateway (correct client per protocol)
+    4. Parses / repairs JSON, validates against the extraction schema
+    5. Records actual lineage (endpoint_id, deployment_id, model, protocol)
+
+    Args:
+        gateway: The shared InferenceGateway instance.
+        target: Resolved inference target for extraction.
+        document_text: The document to extract from.
+        document_type: Type of document (article, filing, transcript, etc.).
+        document_id: UUID of the source document.
+        known_tickers: Optional list of tracked tickers for context.
+        max_retries: Maximum number of retry attempts.
+        retry_base_delay: Initial retry delay in seconds.
+        retry_max_delay: Maximum retry delay in seconds.
+        retry_backoff_multiplier: Backoff multiplier for retries.
+
+    Returns:
+        ExtractionWithLineage containing the extraction response and lineage.
+    """
+    import asyncio
+
+    prompts = build_extraction_prompt(
+        document_text=document_text,
+        document_type=document_type,
+        document_id=document_id,
+        known_tickers=known_tickers,
+    )
+    json_schema = get_json_schema()
+    prompt_meta = get_prompt_metadata()
+
+    response = ExtractionResponse(
+        prompt_metadata=prompt_meta,
+        model=target.model,
+    )
+    inference_results: list[InferenceResult] = []
+    last_lineage: ModelLineage | None = None
+
+    total_start = time.monotonic()
+
+    for attempt_num in range(max_retries + 1):
+        # Build request
+        request = StructuredGenerationRequest(
+            messages=[
+                ChatMessage(role="system", content=prompts["system"]),
+                ChatMessage(role="user", content=prompts["user"]),
+            ],
+            json_schema=json_schema,
+            max_output_tokens=target.extra_body.get("max_tokens", 4096),
+            temperature=0.0,
+            seed=0,
+            timeout_seconds=target.timeout_seconds,
+            trace_id=document_id,
+        )
+
+        # Call via gateway
+        result = await gateway.generate(target, request)
+        inference_results.append(result)
+        last_lineage = build_lineage_from_result(result, trace_id=document_id)
+
+        # Convert to ExtractionAttempt for compatibility
+        attempt = _inference_result_to_attempt(result, target.model, document_text)
+        response.attempts.append(attempt)
+
+        if attempt.error is None and attempt.validation and attempt.validation.valid:
+            response.success = True
+            response.result = attempt.validation.parsed
+            break
+
+        # Determine if retryable
+        retryable = _is_result_retryable(result)
+        attempt.retryable = retryable
+
+        if not retryable:
+            logger.warning(
+                "Non-retryable error for doc %s: %s — stopping retries",
+                document_id or "unknown",
+                attempt.error,
+            )
+            break
+
+        if attempt_num < max_retries:
+            delay = retry_base_delay * (retry_backoff_multiplier ** attempt_num)
+            delay = min(delay, retry_max_delay)
+            logger.warning(
+                "Extraction attempt %d/%d failed for doc %s: %s — retrying in %.1fs",
+                attempt_num + 1,
+                max_retries + 1,
+                document_id or "unknown",
+                attempt.error or "validation failed",
+                delay,
+            )
+            await asyncio.sleep(delay)
+
+    response.total_duration_ms = int((time.monotonic() - total_start) * 1000)
+
+    # Use actual lineage from last inference call
+    lineage = last_lineage or ModelLineage(model=target.model, protocol=target.protocol)
+
+    return ExtractionWithLineage(
+        response=response,
+        lineage=lineage,
+        raw_inference_results=inference_results,
+    )
+
+
+def _inference_result_to_attempt(
+    result: InferenceResult,
+    model: str,
+    document_text: str,
+) -> ExtractionAttempt:
+    """Convert an InferenceResult to the legacy ExtractionAttempt format.
+
+    Applies the same markdown-fence stripping, JSON repair, and schema
+    validation as the existing VLLMClient and OllamaClient.
+    """
+    attempt = ExtractionAttempt(model=model)
+    attempt.duration_ms = result.latency_ms
+    attempt.raw_output = result.content
+
+    # Check for gateway-level errors
+    if result.error:
+        attempt.error = result.error
+        attempt.retryable = _is_result_retryable(result)
+        return attempt
+
+    content = result.content
+    if not content:
+        attempt.error = "empty_model_response"
+        return attempt
+
+    # Strip markdown fences if present
+    content = _strip_markdown_fences(content)
+
+    # Repair malformed JSON
+    content = _repair_json(content)
+
+    # Validate against extraction schema
+    attempt.validation = validate_extraction(content, document_text=document_text)
+    if not attempt.validation.valid:
+        attempt.error = "; ".join(attempt.validation.errors)
+
+    return attempt
+
+
+def _is_result_retryable(result: InferenceResult) -> bool:
+    """Determine if an inference result error is retryable."""
+    if result.error_category in (
+        "timeout",
+        "rate_limit",
+        "server_error",
+        "connection_error",
+    ):
+        return True
+    if result.error and "empty" in result.error.lower():
+        return True
+    return False

services/extractor/main.py

@@ -308,9 +308,20 @@ async def _process_macro_classification(
             len(enqueued_tickers), event.event_id,
         )
 
+        # Update document status to 'extracted' (macro classification is the extraction for these docs)
+        await pool.execute(
+            "UPDATE documents SET status = 'extracted', updated_at = NOW() WHERE id = $1::uuid",
+            document_id,
+        )
+
     except ValueError as e:
         _macro_consecutive_failures += 1
         logger.error("Macro event classification failed for doc %s: %s", document_id, e)
+        # Mark as extraction_failed so it can be retried
+        await pool.execute(
+            "UPDATE documents SET status = 'extraction_failed', updated_at = NOW() WHERE id = $1::uuid",
+            document_id,
+        )
         if _macro_consecutive_failures >= _MACRO_FAILURE_ALERT_THRESHOLD:
             logger.critical(
                 "ALERT: Sustained macro classification failures (%d consecutive). "
@@ -320,6 +331,10 @@ async def _process_macro_classification(
     except Exception:
         _macro_consecutive_failures += 1
         logger.exception("Unexpected error classifying macro event for doc %s", document_id)
+        await pool.execute(
+            "UPDATE documents SET status = 'extraction_failed', updated_at = NOW() WHERE id = $1::uuid",
+            document_id,
+        )
         if _macro_consecutive_failures >= _MACRO_FAILURE_ALERT_THRESHOLD:
             logger.critical(
                 "ALERT: Sustained macro classification failures (%d consecutive). "

services/extractor/schemas.py

@@ -362,15 +362,19 @@ _HORIZON_MAP: dict[str, str] = {
 
 def _normalize_extraction_data(data: dict[str, Any]) -> dict[str, Any]:
     """Fix common model output issues before Pydantic validation."""
-    # Fill missing top-level required fields with defaults
-    data.setdefault("summary", "")
-    data.setdefault("companies", [])
-    data.setdefault("macro_themes", [])
-    if "novelty_score" not in data:
+    # Fill missing or null top-level required fields with defaults
+    if not data.get("summary"):
+        data["summary"] = ""
+    if not isinstance(data.get("companies"), list):
+        data["companies"] = []
+    if not isinstance(data.get("macro_themes"), list):
+        data["macro_themes"] = []
+    if data.get("novelty_score") is None:
         data["novelty_score"] = 0.5
-    if "confidence" not in data:
+    if data.get("confidence") is None:
         data["confidence"] = 0.3
-    data.setdefault("extraction_warnings", ["incomplete_model_output"])
+    if not isinstance(data.get("extraction_warnings"), list):
+        data["extraction_warnings"] = ["incomplete_model_output"]
 
     # Clamp novelty_score and confidence to [0, 1]
     for field in ("novelty_score", "confidence"):
@@ -384,17 +388,22 @@ def _normalize_extraction_data(data: dict[str, Any]) -> dict[str, Any]:
         for comp in companies:
             if not isinstance(comp, dict):
                 continue
-            # Fill missing required company fields with defaults
-            comp.setdefault("ticker", "")
-            comp.setdefault("company_name", "")
-            comp.setdefault("relevance", 0.5)
-            comp.setdefault("sentiment", "neutral")
-            comp.setdefault("impact_score", 0.5)
-            comp.setdefault("impact_horizon", "1d_30d")
-            comp.setdefault("catalyst_type", "other")
-            comp.setdefault("key_facts", [])
-            comp.setdefault("risks", [])
-            comp.setdefault("evidence_spans", [])
+            # Replace None values with defaults (model returns null for fields)
+            _company_defaults = {
+                "ticker": "",
+                "company_name": "",
+                "relevance": 0.5,
+                "sentiment": "neutral",
+                "impact_score": 0.5,
+                "impact_horizon": "1d_30d",
+                "catalyst_type": "other",
+                "key_facts": [],
+                "risks": [],
+                "evidence_spans": [],
+            }
+            for key, default in _company_defaults.items():
+                if comp.get(key) is None:
+                    comp[key] = default
             # Clamp numeric fields
             for f in ("relevance", "impact_score"):
                 v = comp.get(f)
@@ -412,6 +421,13 @@ def _normalize_extraction_data(data: dict[str, Any]) -> dict[str, Any]:
                 mapped_cat = _CATALYST_ALIASES.get(cat.lower().strip(), "other")
                 comp["catalyst_type"] = mapped_cat
 
+        # Filter out company entries with no ticker (model returned null/empty)
+        # — keep the rest of the extraction (summary, macro_themes, etc.)
+        data["companies"] = [
+            c for c in companies
+            if isinstance(c, dict) and c.get("ticker")
+        ]
+
     return data
 
 

services/inference_registry/__init__.py

@@ -0,0 +1,8 @@
+"""Inference Registry API service.
+
+FastAPI router for managing inference_endpoints, model_deployments,
+and agent_stage_bindings. Auth secret values are NEVER returned in
+responses.
+
+Requirements: 3.6, 3.7
+"""

services/inference_registry/router.py

@@ -0,0 +1,634 @@
+"""FastAPI router for the inference registry API.
+
+Manages inference_endpoints, model_deployments, and agent_stage_bindings.
+Auth secret values are NEVER returned in any response.
+
+Endpoints:
+  - CRUD for inference_endpoints (19.1)
+  - probe, enable, disable, test-structured-output actions (19.2)
+  - Protocol/endpoint/deployment selectors (19.3)
+  - Display last probe, capabilities, limits, bindings (19.4)
+  - External egress confirmation (19.5)
+
+Requirements: 3.6, 3.7
+"""
+from __future__ import annotations
+
+import logging
+import uuid
+from datetime import datetime, timezone
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException
+
+from services.inference_registry.schemas import (
+    BindingCreate,
+    BindingResponse,
+    DeploymentCreate,
+    DeploymentResponse,
+    EgressConfirmation,
+    EndpointCreate,
+    EndpointListResponse,
+    EndpointResponse,
+    EndpointUpdate,
+    ProbeResponse,
+    StructuredOutputTestRequest,
+    StructuredOutputTestResponse,
+)
+from services.inference_registry.security import (
+    is_external_endpoint,
+    redact_binding,
+    redact_deployment,
+    redact_endpoint,
+    redact_endpoint_for_list,
+)
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/inference", tags=["inference-registry"])
+
+
+# ---------------------------------------------------------------------------
+# Dependency injection protocol for database access
+# ---------------------------------------------------------------------------
+
+
+class InferenceRegistryDB:
+    """Protocol for inference registry database operations.
+
+    In production, backed by asyncpg pool. In tests, a mock implements this.
+    """
+
+    async def list_endpoints(self) -> list[dict[str, Any]]:
+        raise NotImplementedError
+
+    async def get_endpoint(self, endpoint_id: uuid.UUID) -> dict[str, Any] | None:
+        raise NotImplementedError
+
+    async def create_endpoint(self, data: dict[str, Any]) -> dict[str, Any]:
+        raise NotImplementedError
+
+    async def update_endpoint(
+        self, endpoint_id: uuid.UUID, data: dict[str, Any]
+    ) -> dict[str, Any] | None:
+        raise NotImplementedError
+
+    async def disable_endpoint(self, endpoint_id: uuid.UUID) -> dict[str, Any] | None:
+        raise NotImplementedError
+
+    async def list_deployments(
+        self, endpoint_id: uuid.UUID | None = None
+    ) -> list[dict[str, Any]]:
+        raise NotImplementedError
+
+    async def get_deployment(self, deployment_id: uuid.UUID) -> dict[str, Any] | None:
+        raise NotImplementedError
+
+    async def create_deployment(self, data: dict[str, Any]) -> dict[str, Any]:
+        raise NotImplementedError
+
+    async def list_bindings(
+        self, agent_id: uuid.UUID | None = None,
+        endpoint_id: uuid.UUID | None = None,
+    ) -> list[dict[str, Any]]:
+        raise NotImplementedError
+
+    async def create_binding(self, data: dict[str, Any]) -> dict[str, Any]:
+        raise NotImplementedError
+
+    async def get_bindings_for_endpoint(
+        self, endpoint_id: uuid.UUID
+    ) -> list[dict[str, Any]]:
+        raise NotImplementedError
+
+    async def get_last_probe(
+        self, endpoint_id: uuid.UUID
+    ) -> dict[str, Any] | None:
+        raise NotImplementedError
+
+    async def store_probe_result(
+        self, endpoint_id: uuid.UUID, result: dict[str, Any]
+    ) -> None:
+        raise NotImplementedError
+
+    async def get_egress_confirmation(
+        self, endpoint_id: uuid.UUID
+    ) -> bool:
+        raise NotImplementedError
+
+    async def store_egress_confirmation(
+        self, endpoint_id: uuid.UUID
+    ) -> None:
+        raise NotImplementedError
+
+
+# Global DB instance (set during app startup)
+_db: InferenceRegistryDB | None = None
+
+
+def set_db(db: InferenceRegistryDB) -> None:
+    """Set the database dependency for the router."""
+    global _db
+    _db = db
+
+
+def get_db() -> InferenceRegistryDB:
+    """Get the database dependency."""
+    if _db is None:
+        raise HTTPException(503, "Database not initialized")
+    return _db
+
+
+# ---------------------------------------------------------------------------
+# Endpoint CRUD (19.1)
+# ---------------------------------------------------------------------------
+
+
+@router.get("/endpoints", response_model=list[EndpointListResponse])
+async def list_endpoints(db: InferenceRegistryDB = Depends(get_db)):
+    """List all inference endpoints with secrets redacted."""
+    endpoints = await db.list_endpoints()
+    return [redact_endpoint_for_list(ep) for ep in endpoints]
+
+
+@router.get("/endpoints/{endpoint_id}", response_model=EndpointResponse)
+async def get_endpoint(
+    endpoint_id: uuid.UUID,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Get endpoint detail with secrets redacted.
+
+    Includes last probe results, capabilities, and active stage bindings (19.4).
+    """
+    endpoint = await db.get_endpoint(endpoint_id)
+    if endpoint is None:
+        raise HTTPException(404, "Endpoint not found")
+
+    response = redact_endpoint(endpoint)
+
+    # Attach last probe result (19.4)
+    probe_data = await db.get_last_probe(endpoint_id)
+    if probe_data:
+        response.last_probe = ProbeResponse(**probe_data)
+
+    # Attach capabilities from deployments
+    deployments = await db.list_deployments(endpoint_id=endpoint_id)
+    if deployments:
+        # Aggregate capabilities from all deployments
+        combined_caps: dict[str, Any] = {}
+        for dep in deployments:
+            caps = dep.get("capabilities", {})
+            for k, v in caps.items():
+                if v:
+                    combined_caps[k] = True
+        response.capabilities = combined_caps
+
+    # Attach active bindings (19.4)
+    bindings = await db.get_bindings_for_endpoint(endpoint_id)
+    if bindings:
+        response.active_bindings = [redact_binding(b) for b in bindings]
+
+    return response
+
+
+@router.post("/endpoints", response_model=EndpointResponse, status_code=201)
+async def create_endpoint(
+    body: EndpointCreate,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Create a new inference endpoint.
+
+    Validates protocol and URL format. External endpoints require
+    egress confirmation before they can be enabled (19.5).
+    """
+    now = datetime.now(timezone.utc)
+    endpoint_data = {
+        "id": uuid.uuid4(),
+        "name": body.name,
+        "protocol": body.protocol,
+        "base_url": body.base_url,
+        "auth_secret_ref": body.auth_secret_ref,
+        "auth_scheme": body.auth_scheme,
+        "default_headers": body.default_headers,
+        "health_path": body.health_path,
+        "enabled": body.enabled,
+        "revision": 1,
+        "created_at": now,
+        "updated_at": now,
+    }
+
+    # If external, require egress confirmation before enabling (19.5)
+    if body.enabled and is_external_endpoint(body.base_url):
+        endpoint_data["enabled"] = False  # Will need confirm-egress call
+
+    created = await db.create_endpoint(endpoint_data)
+    return redact_endpoint(created)
+
+
+@router.put("/endpoints/{endpoint_id}", response_model=EndpointResponse)
+async def update_endpoint(
+    endpoint_id: uuid.UUID,
+    body: EndpointUpdate,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Update an existing inference endpoint."""
+    existing = await db.get_endpoint(endpoint_id)
+    if existing is None:
+        raise HTTPException(404, "Endpoint not found")
+
+    update_data: dict[str, Any] = {}
+    for field_name in (
+        "name", "protocol", "base_url", "auth_secret_ref",
+        "auth_scheme", "default_headers", "health_path", "enabled",
+    ):
+        value = getattr(body, field_name)
+        if value is not None:
+            update_data[field_name] = value
+
+    if update_data:
+        update_data["updated_at"] = datetime.now(timezone.utc)
+        update_data["revision"] = existing.get("revision", 1) + 1
+
+    updated = await db.update_endpoint(endpoint_id, update_data)
+    if updated is None:
+        raise HTTPException(404, "Endpoint not found")
+    return redact_endpoint(updated)
+
+
+@router.delete("/endpoints/{endpoint_id}", response_model=EndpointResponse)
+async def delete_endpoint(
+    endpoint_id: uuid.UUID,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Soft-delete (disable) an inference endpoint."""
+    disabled = await db.disable_endpoint(endpoint_id)
+    if disabled is None:
+        raise HTTPException(404, "Endpoint not found")
+    return redact_endpoint(disabled)
+
+
+# ---------------------------------------------------------------------------
+# Endpoint actions (19.2)
+# ---------------------------------------------------------------------------
+
+
+@router.post("/endpoints/{endpoint_id}/probe", response_model=ProbeResponse)
+async def probe_endpoint(
+    endpoint_id: uuid.UUID,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Run a capability probe against the endpoint.
+
+    Performs health check, model listing, JSON Schema test,
+    usage metadata check, seed determinism check, and output-token field check.
+    """
+    endpoint = await db.get_endpoint(endpoint_id)
+    if endpoint is None:
+        raise HTTPException(404, "Endpoint not found")
+
+    # Import the prober
+    from services.shared.inference.capabilities import EndpointProber, FullProbeResult
+    from services.shared.inference.models import InferenceTarget, ProviderCapabilities
+
+    # Get a deployment to probe with (need a model name)
+    deployments = await db.list_deployments(endpoint_id=endpoint_id)
+    model_name = "default"
+    caps_data: dict[str, Any] = {}
+    deployment_id = uuid.uuid4()
+
+    if deployments:
+        model_name = deployments[0].get("served_model_name", "default")
+        caps_data = deployments[0].get("capabilities", {})
+        deployment_id = deployments[0]["id"]
+
+    capabilities = ProviderCapabilities(
+        chat_completions=caps_data.get("chat_completions", True),
+        responses_api=caps_data.get("responses_api", False),
+        json_schema=caps_data.get("json_schema", False),
+        json_object=caps_data.get("json_object", False),
+        seed=caps_data.get("seed", False),
+        usage=caps_data.get("usage", False),
+        max_completion_tokens=caps_data.get("max_completion_tokens", False),
+        reasoning_toggle=caps_data.get("reasoning_toggle", False),
+        model_listing=caps_data.get("model_listing", False),
+    )
+
+    target = InferenceTarget(
+        endpoint_id=endpoint_id,
+        deployment_id=deployment_id,
+        protocol=endpoint["protocol"],
+        base_url=endpoint["base_url"],
+        model=model_name,
+        capabilities=capabilities,
+        auth_secret_ref=endpoint.get("auth_secret_ref"),
+        auth_scheme=endpoint.get("auth_scheme", "bearer"),
+        extra_headers=endpoint.get("default_headers") or {},
+    )
+
+    prober = EndpointProber()
+    try:
+        result: FullProbeResult = await prober.run_full_probe(target)
+    finally:
+        await prober.close()
+
+    # Build probe response
+    probe_response = ProbeResponse(
+        endpoint_id=endpoint_id,
+        timestamp=result.timestamp,
+        software_version=result.software_version,
+        probe_duration_ms=result.probe_duration_ms,
+        health_success=result.health.success if result.health else False,
+        health_detail=result.health.detail if result.health else "",
+        model_listing_success=(
+            result.model_listing.success if result.model_listing else None
+        ),
+        json_schema_success=(
+            result.json_schema.success if result.json_schema else None
+        ),
+        usage_success=(
+            result.usage_metadata.success if result.usage_metadata else None
+        ),
+        seed_success=(
+            result.seed_determinism.success if result.seed_determinism else None
+        ),
+        output_token_field_success=(
+            result.output_token_field.success if result.output_token_field else None
+        ),
+    )
+
+    # Store probe result
+    await db.store_probe_result(endpoint_id, probe_response.model_dump())
+
+    return probe_response
+
+
+@router.post("/endpoints/{endpoint_id}/enable", response_model=EndpointResponse)
+async def enable_endpoint(
+    endpoint_id: uuid.UUID,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Enable an inference endpoint.
+
+    External endpoints require egress confirmation first (19.5).
+    """
+    endpoint = await db.get_endpoint(endpoint_id)
+    if endpoint is None:
+        raise HTTPException(404, "Endpoint not found")
+
+    # Check if external endpoint needs egress confirmation (19.5)
+    if is_external_endpoint(endpoint["base_url"]):
+        has_confirmation = await db.get_egress_confirmation(endpoint_id)
+        if not has_confirmation:
+            raise HTTPException(
+                403,
+                "External endpoint requires egress confirmation. "
+                "POST /api/inference/endpoints/{id}/confirm-egress first.",
+            )
+
+    update_data = {
+        "enabled": True,
+        "updated_at": datetime.now(timezone.utc),
+        "revision": endpoint.get("revision", 1) + 1,
+    }
+    updated = await db.update_endpoint(endpoint_id, update_data)
+    if updated is None:
+        raise HTTPException(404, "Endpoint not found")
+    return redact_endpoint(updated)
+
+
+@router.post("/endpoints/{endpoint_id}/disable", response_model=EndpointResponse)
+async def disable_endpoint_action(
+    endpoint_id: uuid.UUID,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Disable an inference endpoint."""
+    endpoint = await db.get_endpoint(endpoint_id)
+    if endpoint is None:
+        raise HTTPException(404, "Endpoint not found")
+
+    update_data = {
+        "enabled": False,
+        "updated_at": datetime.now(timezone.utc),
+        "revision": endpoint.get("revision", 1) + 1,
+    }
+    updated = await db.update_endpoint(endpoint_id, update_data)
+    if updated is None:
+        raise HTTPException(404, "Endpoint not found")
+    return redact_endpoint(updated)
+
+
+@router.post(
+    "/endpoints/{endpoint_id}/test-structured-output",
+    response_model=StructuredOutputTestResponse,
+)
+async def test_structured_output(
+    endpoint_id: uuid.UUID,
+    body: StructuredOutputTestRequest | None = None,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Test JSON Schema structured output on an endpoint.
+
+    Sends a minimal schema-constrained request and validates the response.
+    """
+    if body is None:
+        body = StructuredOutputTestRequest()
+
+    endpoint = await db.get_endpoint(endpoint_id)
+    if endpoint is None:
+        raise HTTPException(404, "Endpoint not found")
+
+    from services.shared.inference.capabilities import EndpointProber
+    from services.shared.inference.models import InferenceTarget, ProviderCapabilities
+
+    # Get a deployment to test with
+    deployments = await db.list_deployments(endpoint_id=endpoint_id)
+    model_name = "default"
+    deployment_id = uuid.uuid4()
+    if deployments:
+        model_name = deployments[0].get("served_model_name", "default")
+        deployment_id = deployments[0]["id"]
+
+    target = InferenceTarget(
+        endpoint_id=endpoint_id,
+        deployment_id=deployment_id,
+        protocol=endpoint["protocol"],
+        base_url=endpoint["base_url"],
+        model=model_name,
+        capabilities=ProviderCapabilities(
+            chat_completions=True,
+            json_schema=True,
+        ),
+        auth_secret_ref=endpoint.get("auth_secret_ref"),
+        auth_scheme=endpoint.get("auth_scheme", "bearer"),
+        extra_headers=endpoint.get("default_headers") or {},
+    )
+
+    prober = EndpointProber()
+    try:
+        result = await prober.probe_json_schema(target)
+    finally:
+        await prober.close()
+
+    return StructuredOutputTestResponse(
+        success=result.success,
+        structured_mode=result.structured_mode_used,
+        content=result.detail,
+        schema_valid=result.schema_valid,
+        error=None if result.success else result.detail,
+    )
+
+
+# ---------------------------------------------------------------------------
+# External egress confirmation (19.5)
+# ---------------------------------------------------------------------------
+
+
+@router.post("/endpoints/{endpoint_id}/confirm-egress", response_model=EndpointResponse)
+async def confirm_egress(
+    endpoint_id: uuid.UUID,
+    body: EgressConfirmation,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Confirm external endpoint egress enablement.
+
+    Required before enabling an endpoint with a non-cluster URL.
+    The request body must contain {"confirmed": true}.
+    """
+    endpoint = await db.get_endpoint(endpoint_id)
+    if endpoint is None:
+        raise HTTPException(404, "Endpoint not found")
+
+    if not is_external_endpoint(endpoint["base_url"]):
+        raise HTTPException(400, "Endpoint is not external; no egress confirmation needed")
+
+    # body.confirmed is already validated by pydantic to be True
+    await db.store_egress_confirmation(endpoint_id)
+
+    # Now enable the endpoint
+    update_data = {
+        "enabled": True,
+        "updated_at": datetime.now(timezone.utc),
+        "revision": endpoint.get("revision", 1) + 1,
+    }
+    updated = await db.update_endpoint(endpoint_id, update_data)
+    if updated is None:
+        raise HTTPException(404, "Endpoint not found")
+    return redact_endpoint(updated)
+
+
+# ---------------------------------------------------------------------------
+# Deployments (19.3, 19.4)
+# ---------------------------------------------------------------------------
+
+
+@router.get("/deployments", response_model=list[DeploymentResponse])
+async def list_deployments(
+    endpoint_id: uuid.UUID | None = None,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """List model deployments, optionally filtered by endpoint."""
+    deployments = await db.list_deployments(endpoint_id=endpoint_id)
+    return [redact_deployment(d) for d in deployments]
+
+
+@router.get("/deployments/{deployment_id}", response_model=DeploymentResponse)
+async def get_deployment(
+    deployment_id: uuid.UUID,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Get a model deployment with capabilities and limits (19.4)."""
+    deployment = await db.get_deployment(deployment_id)
+    if deployment is None:
+        raise HTTPException(404, "Deployment not found")
+    return redact_deployment(deployment)
+
+
+@router.post("/deployments", response_model=DeploymentResponse, status_code=201)
+async def create_deployment(
+    body: DeploymentCreate,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Create a new model deployment."""
+    # Verify endpoint exists
+    endpoint = await db.get_endpoint(body.endpoint_id)
+    if endpoint is None:
+        raise HTTPException(404, "Referenced endpoint not found")
+
+    deployment_data = {
+        "id": uuid.uuid4(),
+        "endpoint_id": body.endpoint_id,
+        "served_model_name": body.served_model_name,
+        "display_name": body.display_name,
+        "capabilities": body.capabilities,
+        "context_window": body.context_window,
+        "max_output_tokens": body.max_output_tokens,
+        "quantization": body.quantization,
+        "runtime_metadata": body.runtime_metadata,
+        "enabled": body.enabled,
+        "revision": 1,
+    }
+
+    created = await db.create_deployment(deployment_data)
+    return redact_deployment(created)
+
+
+# ---------------------------------------------------------------------------
+# Bindings (19.3, 19.4)
+# ---------------------------------------------------------------------------
+
+
+@router.get("/bindings", response_model=list[BindingResponse])
+async def list_bindings(
+    agent_id: uuid.UUID | None = None,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """List agent stage bindings."""
+    bindings = await db.list_bindings(agent_id=agent_id)
+    return [redact_binding(b) for b in bindings]
+
+
+@router.post("/bindings", response_model=BindingResponse, status_code=201)
+async def create_binding(
+    body: BindingCreate,
+    db: InferenceRegistryDB = Depends(get_db),
+):
+    """Create an agent stage binding."""
+    # Verify deployment exists if provided
+    if body.model_deployment_id:
+        deployment = await db.get_deployment(body.model_deployment_id)
+        if deployment is None:
+            raise HTTPException(404, "Referenced deployment not found")
+
+    binding_data = {
+        "id": uuid.uuid4(),
+        "agent_id": body.agent_id,
+        "stage": body.stage,
+        "model_deployment_id": body.model_deployment_id,
+        "route_order": body.route_order,
+        "routing_config": body.routing_config,
+        "is_active": body.is_active,
+        "revision": 1,
+    }
+
+    created = await db.create_binding(binding_data)
+    return redact_binding(created)
+
+
+# ---------------------------------------------------------------------------
+# Selectors (19.3)
+# ---------------------------------------------------------------------------
+
+
+@router.get("/protocols")
+async def list_protocols():
+    """Return available protocol options for endpoint creation.
+
+    Replaces free-text provider inputs with controlled selectors.
+    """
+    return {
+        "protocols": [
+            {"value": "ollama_native", "label": "Ollama Native", "description": "Ollama /api/chat endpoint"},
+            {"value": "openai_chat", "label": "OpenAI Compatible", "description": "OpenAI /v1/chat/completions (vLLM, OpenAI, LM Studio, SGLang)"},
+            {"value": "specialist_http", "label": "Specialist HTTP", "description": "Typed non-generative endpoints (GLiNER, FinBERT)"},
+        ]
+    }

services/inference_registry/schemas.py

@@ -0,0 +1,251 @@
+"""Pydantic request/response models for the inference registry API.
+
+All response models EXCLUDE actual auth_secret_ref values.
+Instead they show a status string: "configured" or "not_configured".
+
+Requirements: 3.6, 3.7
+"""
+from __future__ import annotations
+
+from datetime import datetime
+from typing import Any, Literal
+from uuid import UUID
+
+from pydantic import BaseModel, Field, field_validator
+
+VALID_PROTOCOLS = ("ollama_native", "openai_chat", "specialist_http")
+
+
+# ---------------------------------------------------------------------------
+# Endpoint schemas
+# ---------------------------------------------------------------------------
+
+
+class EndpointCreate(BaseModel):
+    """Request body for creating an inference endpoint."""
+
+    name: str = Field(..., min_length=1, max_length=255)
+    protocol: Literal["ollama_native", "openai_chat", "specialist_http"]
+    base_url: str = Field(..., min_length=1)
+    auth_secret_ref: str | None = None
+    auth_scheme: str = "bearer"
+    default_headers: dict[str, str] = Field(default_factory=dict)
+    health_path: str | None = None
+    enabled: bool = True
+
+    @field_validator("base_url")
+    @classmethod
+    def validate_url(cls, v: str) -> str:
+        """Validate that base_url looks like a valid URL."""
+        if not v.startswith(("http://", "https://")):
+            raise ValueError("base_url must start with http:// or https://")
+        return v.rstrip("/")
+
+    @field_validator("protocol")
+    @classmethod
+    def validate_protocol(cls, v: str) -> str:
+        if v not in VALID_PROTOCOLS:
+            raise ValueError(f"protocol must be one of {VALID_PROTOCOLS}")
+        return v
+
+
+class EndpointUpdate(BaseModel):
+    """Request body for updating an inference endpoint."""
+
+    name: str | None = None
+    protocol: Literal["ollama_native", "openai_chat", "specialist_http"] | None = None
+    base_url: str | None = None
+    auth_secret_ref: str | None = Field(default=None)
+    auth_scheme: str | None = None
+    default_headers: dict[str, str] | None = None
+    health_path: str | None = None
+    enabled: bool | None = None
+
+    @field_validator("base_url")
+    @classmethod
+    def validate_url(cls, v: str | None) -> str | None:
+        if v is not None:
+            if not v.startswith(("http://", "https://")):
+                raise ValueError("base_url must start with http:// or https://")
+            return v.rstrip("/")
+        return v
+
+    @field_validator("protocol")
+    @classmethod
+    def validate_protocol(cls, v: str | None) -> str | None:
+        if v is not None and v not in VALID_PROTOCOLS:
+            raise ValueError(f"protocol must be one of {VALID_PROTOCOLS}")
+        return v
+
+
+class EndpointResponse(BaseModel):
+    """Response model for an inference endpoint. NEVER includes auth_secret_ref value."""
+
+    id: UUID
+    name: str
+    protocol: str
+    base_url: str
+    auth_secret_status: str = "not_configured"  # "configured" or "not_configured"
+    auth_scheme: str = "bearer"
+    default_headers: dict[str, str] = Field(default_factory=dict)
+    health_path: str | None = None
+    enabled: bool = True
+    revision: int = 1
+    created_at: datetime | None = None
+    updated_at: datetime | None = None
+    last_probe: ProbeResponse | None = None
+    capabilities: dict[str, Any] | None = None
+    active_bindings: list[BindingResponse] | None = None
+
+
+class EndpointListResponse(BaseModel):
+    """Response model for listing endpoints."""
+
+    id: UUID
+    name: str
+    protocol: str
+    base_url: str
+    auth_secret_status: str = "not_configured"
+    enabled: bool = True
+    revision: int = 1
+    created_at: datetime | None = None
+    updated_at: datetime | None = None
+
+
+# ---------------------------------------------------------------------------
+# Deployment schemas
+# ---------------------------------------------------------------------------
+
+
+class DeploymentCreate(BaseModel):
+    """Request body for creating a model deployment."""
+
+    endpoint_id: UUID
+    served_model_name: str = Field(..., min_length=1)
+    display_name: str = Field(..., min_length=1)
+    capabilities: dict[str, Any] = Field(default_factory=dict)
+    context_window: int | None = None
+    max_output_tokens: int | None = None
+    quantization: str | None = None
+    runtime_metadata: dict[str, Any] = Field(default_factory=dict)
+    enabled: bool = True
+
+
+class DeploymentResponse(BaseModel):
+    """Response model for a model deployment."""
+
+    id: UUID
+    endpoint_id: UUID
+    served_model_name: str
+    display_name: str
+    capabilities: dict[str, Any] = Field(default_factory=dict)
+    context_window: int | None = None
+    max_output_tokens: int | None = None
+    quantization: str | None = None
+    runtime_metadata: dict[str, Any] = Field(default_factory=dict)
+    enabled: bool = True
+    revision: int = 1
+
+
+# ---------------------------------------------------------------------------
+# Binding schemas
+# ---------------------------------------------------------------------------
+
+
+class BindingCreate(BaseModel):
+    """Request body for creating an agent stage binding."""
+
+    agent_id: UUID
+    stage: str = Field(..., min_length=1)
+    model_deployment_id: UUID | None = None
+    route_order: int = 0
+    routing_config: dict[str, Any] = Field(default_factory=dict)
+    is_active: bool = True
+
+
+class BindingResponse(BaseModel):
+    """Response model for an agent stage binding."""
+
+    id: UUID
+    agent_id: UUID
+    stage: str
+    model_deployment_id: UUID | None = None
+    route_order: int = 0
+    routing_config: dict[str, Any] = Field(default_factory=dict)
+    is_active: bool = True
+    revision: int = 1
+
+
+# ---------------------------------------------------------------------------
+# Probe schemas
+# ---------------------------------------------------------------------------
+
+
+class ProbeResponse(BaseModel):
+    """Response model for probe results."""
+
+    endpoint_id: UUID
+    timestamp: datetime | None = None
+    software_version: str | None = None
+    probe_duration_ms: int = 0
+    health_success: bool = False
+    health_detail: str = ""
+    model_listing_success: bool | None = None
+    json_schema_success: bool | None = None
+    usage_success: bool | None = None
+    seed_success: bool | None = None
+    output_token_field_success: bool | None = None
+
+
+# ---------------------------------------------------------------------------
+# Egress confirmation schema
+# ---------------------------------------------------------------------------
+
+
+class EgressConfirmation(BaseModel):
+    """Request body for confirming external endpoint egress enablement.
+
+    Requires explicit confirmed=true flag.
+    """
+
+    confirmed: bool = Field(
+        ...,
+        description="Must be explicitly set to true to confirm external egress enablement",
+    )
+
+    @field_validator("confirmed")
+    @classmethod
+    def must_be_true(cls, v: bool) -> bool:
+        if not v:
+            raise ValueError("confirmed must be true to enable external egress")
+        return v
+
+
+# ---------------------------------------------------------------------------
+# Structured output test schemas
+# ---------------------------------------------------------------------------
+
+
+class StructuredOutputTestRequest(BaseModel):
+    """Request body for testing structured output on an endpoint."""
+
+    json_schema: dict[str, Any] = Field(
+        default_factory=lambda: {
+            "type": "object",
+            "properties": {"status": {"type": "string"}},
+            "required": ["status"],
+        }
+    )
+    prompt: str = 'Respond with JSON: {"status": "ok"}'
+
+
+class StructuredOutputTestResponse(BaseModel):
+    """Response for structured output test."""
+
+    success: bool
+    structured_mode: str = ""
+    content: str = ""
+    parsed: dict[str, Any] | None = None
+    schema_valid: bool = False
+    latency_ms: int = 0
+    error: str | None = None

services/inference_registry/security.py

@@ -0,0 +1,129 @@
+"""Security helpers for the inference registry API.
+
+Ensures auth_secret_ref values are NEVER exposed in API responses.
+Replaces the actual secret reference with a status string.
+
+Requirements: 3.6
+"""
+from __future__ import annotations
+
+from typing import Any
+
+from services.inference_registry.schemas import (
+    BindingResponse,
+    DeploymentResponse,
+    EndpointListResponse,
+    EndpointResponse,
+)
+
+
+def redact_endpoint(endpoint_dict: dict[str, Any]) -> EndpointResponse:
+    """Convert a raw endpoint dict to an EndpointResponse with secrets redacted.
+
+    The auth_secret_ref value is replaced with a status string:
+    - "configured" if a secret reference exists
+    - "not_configured" if no secret reference is set
+
+    The actual secret ref value is NEVER included in the response.
+    """
+    auth_secret_ref = endpoint_dict.get("auth_secret_ref")
+    auth_secret_status = "configured" if auth_secret_ref else "not_configured"
+
+    return EndpointResponse(
+        id=endpoint_dict["id"],
+        name=endpoint_dict["name"],
+        protocol=endpoint_dict["protocol"],
+        base_url=endpoint_dict["base_url"],
+        auth_secret_status=auth_secret_status,
+        auth_scheme=endpoint_dict.get("auth_scheme", "bearer"),
+        default_headers=endpoint_dict.get("default_headers") or {},
+        health_path=endpoint_dict.get("health_path"),
+        enabled=endpoint_dict.get("enabled", True),
+        revision=endpoint_dict.get("revision", 1),
+        created_at=endpoint_dict.get("created_at"),
+        updated_at=endpoint_dict.get("updated_at"),
+    )
+
+
+def redact_endpoint_for_list(endpoint_dict: dict[str, Any]) -> EndpointListResponse:
+    """Convert a raw endpoint dict to a list response with secrets redacted."""
+    auth_secret_ref = endpoint_dict.get("auth_secret_ref")
+    auth_secret_status = "configured" if auth_secret_ref else "not_configured"
+
+    return EndpointListResponse(
+        id=endpoint_dict["id"],
+        name=endpoint_dict["name"],
+        protocol=endpoint_dict["protocol"],
+        base_url=endpoint_dict["base_url"],
+        auth_secret_status=auth_secret_status,
+        enabled=endpoint_dict.get("enabled", True),
+        revision=endpoint_dict.get("revision", 1),
+        created_at=endpoint_dict.get("created_at"),
+        updated_at=endpoint_dict.get("updated_at"),
+    )
+
+
+def redact_deployment(deployment_dict: dict[str, Any]) -> DeploymentResponse:
+    """Convert a raw deployment dict to a DeploymentResponse."""
+    return DeploymentResponse(
+        id=deployment_dict["id"],
+        endpoint_id=deployment_dict["endpoint_id"],
+        served_model_name=deployment_dict["served_model_name"],
+        display_name=deployment_dict["display_name"],
+        capabilities=deployment_dict.get("capabilities") or {},
+        context_window=deployment_dict.get("context_window"),
+        max_output_tokens=deployment_dict.get("max_output_tokens"),
+        quantization=deployment_dict.get("quantization"),
+        runtime_metadata=deployment_dict.get("runtime_metadata") or {},
+        enabled=deployment_dict.get("enabled", True),
+        revision=deployment_dict.get("revision", 1),
+    )
+
+
+def redact_binding(binding_dict: dict[str, Any]) -> BindingResponse:
+    """Convert a raw binding dict to a BindingResponse."""
+    return BindingResponse(
+        id=binding_dict["id"],
+        agent_id=binding_dict["agent_id"],
+        stage=binding_dict["stage"],
+        model_deployment_id=binding_dict.get("model_deployment_id"),
+        route_order=binding_dict.get("route_order", 0),
+        routing_config=binding_dict.get("routing_config") or {},
+        is_active=binding_dict.get("is_active", True),
+        revision=binding_dict.get("revision", 1),
+    )
+
+
+def is_external_endpoint(base_url: str) -> bool:
+    """Determine if an endpoint URL points to an external (non-cluster) service.
+
+    External endpoints require egress confirmation before enablement.
+    Local/cluster endpoints match:
+    - localhost / 127.0.0.1
+    - *.svc.cluster.local (Kubernetes internal)
+    - 10.x.x.x / 192.168.x.x (private network)
+    """
+    from urllib.parse import urlparse
+
+    parsed = urlparse(base_url)
+    hostname = parsed.hostname or ""
+
+    # Cluster-local patterns
+    if hostname in ("localhost", "127.0.0.1", "::1"):
+        return False
+    if hostname.endswith(".svc.cluster.local"):
+        return False
+    if hostname.startswith("10.") or hostname.startswith("192.168."):
+        return False
+    # Additional private ranges
+    if hostname.startswith("172."):
+        parts = hostname.split(".")
+        if len(parts) >= 2:
+            try:
+                second = int(parts[1])
+                if 16 <= second <= 31:
+                    return False
+            except ValueError:
+                pass
+
+    return True

services/intelligence_pipeline_v3/__init__.py

@@ -0,0 +1 @@
+"""Intelligence Pipeline v3 — staged evidence-grounded extraction architecture."""

services/intelligence_pipeline_v3/active_learning/__init__.py

@@ -0,0 +1,20 @@
+"""Active learning export module.
+
+Selects low-confidence, conflicting, adjudicated, and corrected cases
+for training data. Applies policy filtering for sensitive content and
+exports in a versioned format with full provenance.
+"""
+
+from services.intelligence_pipeline_v3.active_learning.exporter import (
+    ActiveLearningExporter,
+    ExportConfig,
+    ExportRecord,
+    SelectionCriteria,
+)
+
+__all__ = [
+    "ActiveLearningExporter",
+    "ExportConfig",
+    "ExportRecord",
+    "SelectionCriteria",
+]

services/intelligence_pipeline_v3/active_learning/exporter.py

@@ -0,0 +1,204 @@
+"""Active learning data exporter.
+
+Selects training examples from low-confidence, conflicting, adjudicated,
+and reviewer-corrected cases. Applies content policy filters and exports
+in a versioned format with source spans, labels, relations, decisions,
+and provenance.
+"""
+
+from __future__ import annotations
+
+import enum
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID, uuid4
+
+
+class SelectionCriteria(str, enum.Enum):
+    """Why a case was selected for active learning."""
+
+    LOW_CONFIDENCE = "low_confidence"
+    CONFLICTING = "conflicting"
+    ADJUDICATED = "adjudicated"
+    REVIEWER_CORRECTED = "reviewer_corrected"
+    HIGH_DISAGREEMENT = "high_disagreement"
+    NOVEL_PATTERN = "novel_pattern"
+
+
+class ContentPolicy(str, enum.Enum):
+    """Content policy levels for export filtering."""
+
+    ALLOW = "allow"
+    REDACT_PII = "redact_pii"
+    EXCLUDE = "exclude"
+
+
+@dataclass(frozen=True)
+class ExportRecord:
+    """A single active-learning export record.
+
+    Contains source text spans, schema labels, relations, adjudicator
+    decisions, reviewer corrections, and full provenance.
+    """
+
+    record_id: UUID
+    document_id: str
+    selection_criteria: SelectionCriteria
+    export_version: str
+    timestamp: datetime
+
+    # Source content
+    source_spans: list[dict[str, Any]]  # text, start, end, chunk_id
+    document_type: str = ""
+
+    # Labels and annotations
+    entity_labels: list[dict[str, Any]] = field(default_factory=list)
+    relation_labels: list[dict[str, Any]] = field(default_factory=list)
+    event_labels: list[dict[str, Any]] = field(default_factory=list)
+    fact_labels: list[dict[str, Any]] = field(default_factory=list)
+
+    # Decisions and corrections
+    adjudicator_decisions: list[dict[str, Any]] = field(default_factory=list)
+    reviewer_corrections: list[dict[str, Any]] = field(default_factory=list)
+
+    # Provenance
+    pipeline_run_id: UUID | None = None
+    model_versions: dict[str, str] = field(default_factory=dict)
+    confidence_scores: dict[str, float] = field(default_factory=dict)
+
+
+@dataclass
+class ExportConfig:
+    """Configuration for active learning export."""
+
+    export_version: str = "1.0"
+    min_confidence_threshold: float = 0.5  # Select cases below this
+    max_export_count: int = 1000
+    include_adjudicated: bool = True
+    include_corrections: bool = True
+    include_low_confidence: bool = True
+    include_conflicting: bool = True
+    content_policy: ContentPolicy = ContentPolicy.REDACT_PII
+    excluded_fields: set[str] = field(default_factory=set)
+    sensitive_patterns: list[str] = field(default_factory=list)
+
+
+@dataclass
+class ActiveLearningExporter:
+    """Exports selected cases for specialist model training.
+
+    Applies selection criteria, content policy filtering, and
+    produces versioned export datasets with full provenance.
+    """
+
+    config: ExportConfig
+    _records: list[ExportRecord] = field(default_factory=list)
+    _excluded_count: int = 0
+
+    def select_record(
+        self,
+        document_id: str,
+        criteria: SelectionCriteria,
+        source_spans: list[dict[str, Any]],
+        document_type: str = "",
+        entity_labels: list[dict[str, Any]] | None = None,
+        relation_labels: list[dict[str, Any]] | None = None,
+        event_labels: list[dict[str, Any]] | None = None,
+        fact_labels: list[dict[str, Any]] | None = None,
+        adjudicator_decisions: list[dict[str, Any]] | None = None,
+        reviewer_corrections: list[dict[str, Any]] | None = None,
+        pipeline_run_id: UUID | None = None,
+        model_versions: dict[str, str] | None = None,
+        confidence_scores: dict[str, float] | None = None,
+    ) -> ExportRecord | None:
+        """Select a case for export, applying content policy.
+
+        Returns None if the case is excluded by policy.
+        """
+        if len(self._records) >= self.config.max_export_count:
+            return None
+
+        # Apply content policy
+        filtered_spans = self._apply_content_policy(source_spans)
+        if not filtered_spans:
+            self._excluded_count += 1
+            return None
+
+        record = ExportRecord(
+            record_id=uuid4(),
+            document_id=document_id,
+            selection_criteria=criteria,
+            export_version=self.config.export_version,
+            timestamp=datetime.now(timezone.utc),
+            source_spans=filtered_spans,
+            document_type=document_type,
+            entity_labels=entity_labels or [],
+            relation_labels=relation_labels or [],
+            event_labels=event_labels or [],
+            fact_labels=fact_labels or [],
+            adjudicator_decisions=adjudicator_decisions or [],
+            reviewer_corrections=reviewer_corrections or [],
+            pipeline_run_id=pipeline_run_id,
+            model_versions=model_versions or {},
+            confidence_scores=confidence_scores or {},
+        )
+        self._records.append(record)
+        return record
+
+    def _apply_content_policy(
+        self, spans: list[dict[str, Any]]
+    ) -> list[dict[str, Any]]:
+        """Apply content policy filtering to source spans."""
+        if self.config.content_policy == ContentPolicy.EXCLUDE:
+            # Check for sensitive content
+            for span in spans:
+                text = span.get("text", "")
+                if self._contains_sensitive(text):
+                    return []  # Exclude entire record
+
+        if self.config.content_policy == ContentPolicy.REDACT_PII:
+            return [self._redact_span(span) for span in spans]
+
+        return spans
+
+    def _contains_sensitive(self, text: str) -> bool:
+        """Check if text contains sensitive content per policy."""
+        for pattern in self.config.sensitive_patterns:
+            if pattern.lower() in text.lower():
+                return True
+        return False
+
+    def _redact_span(self, span: dict[str, Any]) -> dict[str, Any]:
+        """Redact PII from a span while preserving structure."""
+        # In production, this would use NER-based PII detection
+        # For now, preserve the span but mark it as redacted if needed
+        return {**span, "content_policy_applied": "redact_pii"}
+
+    @property
+    def records(self) -> list[ExportRecord]:
+        return list(self._records)
+
+    @property
+    def total_exported(self) -> int:
+        return len(self._records)
+
+    @property
+    def total_excluded(self) -> int:
+        return self._excluded_count
+
+    def export_manifest(self) -> dict[str, Any]:
+        """Generate export manifest with metadata."""
+        criteria_counts: dict[str, int] = {}
+        for r in self._records:
+            key = r.selection_criteria.value
+            criteria_counts[key] = criteria_counts.get(key, 0) + 1
+
+        return {
+            "export_version": self.config.export_version,
+            "exported_at": datetime.now(timezone.utc).isoformat(),
+            "total_records": len(self._records),
+            "excluded_count": self._excluded_count,
+            "content_policy": self.config.content_policy.value,
+            "selection_criteria_distribution": criteria_counts,
+        }

services/intelligence_pipeline_v3/adjudication/__init__.py

@@ -0,0 +1,60 @@
+"""Adjudication layer for Intelligence Pipeline v3.
+
+This package provides:
+- Schemas for adjudication candidates, conflicts, evidence, questions, and decisions
+- Focused adjudication prompt building with strict JSON Schema output
+- 9B adjudicator deployment configuration and VRAM gating
+- Post-adjudication verification ensuring evidence grounding
+"""
+
+from services.intelligence_pipeline_v3.adjudication.deployment import (
+    APPROVED_MODEL,
+    APPROVED_VLLM_VERSION,
+    AlertConfig,
+    ConcurrencySemaphore,
+    check_vram_gate,
+    verify_structured_output,
+)
+from services.intelligence_pipeline_v3.adjudication.prompts import (
+    AdjudicationPacket,
+    PromptMetadata,
+    build_adjudication_packet,
+)
+from services.intelligence_pipeline_v3.adjudication.schemas import (
+    AdjudicationCandidate,
+    AdjudicationDecision,
+    AdjudicationQuestion,
+    ConflictDescription,
+    EvidencePacket,
+)
+from services.intelligence_pipeline_v3.adjudication.verification import (
+    AdjudicationRecord,
+    RejectionResult,
+    preserve_pre_and_post,
+    reject_unsupported_decisions,
+    route_repeated_failures,
+    verify_evidence_references,
+)
+
+__all__ = [
+    "APPROVED_MODEL",
+    "APPROVED_VLLM_VERSION",
+    "AdjudicationCandidate",
+    "AdjudicationDecision",
+    "AdjudicationPacket",
+    "AdjudicationQuestion",
+    "AdjudicationRecord",
+    "AlertConfig",
+    "ConcurrencySemaphore",
+    "ConflictDescription",
+    "EvidencePacket",
+    "PromptMetadata",
+    "RejectionResult",
+    "build_adjudication_packet",
+    "check_vram_gate",
+    "preserve_pre_and_post",
+    "reject_unsupported_decisions",
+    "route_repeated_failures",
+    "verify_evidence_references",
+    "verify_structured_output",
+]

services/intelligence_pipeline_v3/adjudication/deployment.py

@@ -0,0 +1,180 @@
+"""9B adjudicator deployment configuration for Intelligence Pipeline v3.
+
+Manages the approved model/version pins, VRAM gating, concurrency
+semaphore configuration, and alerting thresholds for the 9B adjudicator
+running on RTX 4070 Ti SUPER via vLLM.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+# --- Pinned model and version constants ---
+
+APPROVED_MODEL: str = "AxionML/Qwen3.5-9B-NVFP4"
+"""Approved 9B model for adjudication. NVFP4 quantization for 4070 Ti SUPER."""
+
+APPROVED_VLLM_VERSION: str = "0.8.5"
+"""Approved vLLM version matching the cluster deployment."""
+
+APPROVED_SERVED_NAME: str = "stonks-adjudicator-9b"
+"""The served model name exposed by the vLLM deployment."""
+
+MAX_MODEL_LEN: int = 8192
+"""Maximum model context length configured for the deployment."""
+
+MAX_NUM_SEQS: int = 8
+"""Maximum concurrent sequences for the vLLM deployment."""
+
+GPU_MEMORY_UTILIZATION: float = 0.80
+"""Target GPU memory utilization fraction."""
+
+VRAM_GATE_PERCENT: float = 5.0
+"""Maximum allowed VRAM increase over baseline (percentage)."""
+
+
+# --- Structured output verification ---
+
+
+def verify_structured_output(target: dict[str, Any]) -> bool:
+    """Verify that structured output works with the given deployment target.
+
+    Checks that the target deployment declares json_schema support in its
+    capabilities and that the required configuration fields are present.
+
+    Args:
+        target: Deployment target configuration dict containing at minimum:
+            - capabilities: dict with json_schema boolean
+            - served_model_name: str matching APPROVED_SERVED_NAME
+            - vllm_version: str for version verification
+
+    Returns:
+        True if strict schema output is expected to work, False otherwise.
+    """
+    capabilities = target.get("capabilities", {})
+    if not capabilities.get("json_schema", False):
+        return False
+
+    # Verify model matches approved deployment
+    served_name = target.get("served_model_name", "")
+    if served_name and served_name != APPROVED_SERVED_NAME:
+        return False
+
+    # Verify vLLM version compatibility
+    vllm_version = target.get("vllm_version", "")
+    if vllm_version and vllm_version != APPROVED_VLLM_VERSION:
+        return False
+
+    # Verify the model is the approved one
+    model = target.get("model", "")
+    if model and model != APPROVED_MODEL:
+        return False
+
+    return True
+
+
+# --- VRAM gate ---
+
+
+def check_vram_gate(peak_mb: float, baseline_mb: float) -> bool:
+    """Check whether peak VRAM usage is within the +5% gate of baseline.
+
+    The gate ensures that no deployment update exceeds the measured current
+    9B deployment VRAM by more than 5 percent.
+
+    Args:
+        peak_mb: Measured peak VRAM in megabytes during test.
+        baseline_mb: Baseline VRAM measurement in megabytes.
+
+    Returns:
+        True if peak is within acceptable range, False if it exceeds the gate.
+    """
+    if baseline_mb <= 0:
+        return False
+    if peak_mb <= 0:
+        return False
+
+    max_allowed_mb = baseline_mb * (1.0 + VRAM_GATE_PERCENT / 100.0)
+    return peak_mb <= max_allowed_mb
+
+
+# --- Concurrency semaphore ---
+
+
+class ConcurrencySemaphore(BaseModel):
+    """Configuration for the adjudication concurrency semaphore.
+
+    Limits concurrent adjudication requests to protect vLLM from
+    overload. Aligned with max-num-seqs and KV-cache behavior.
+    """
+
+    max_concurrent: int = Field(
+        default=MAX_NUM_SEQS,
+        gt=0,
+        description="Maximum concurrent adjudication requests",
+    )
+    queue_timeout_seconds: float = Field(
+        default=120.0,
+        gt=0,
+        description="Maximum time to wait for semaphore acquisition",
+    )
+    backpressure_threshold: int = Field(
+        default=MAX_NUM_SEQS * 4,
+        ge=0,
+        description="Queue depth at which backpressure signals are emitted",
+    )
+
+    def create_semaphore(self) -> asyncio.Semaphore:
+        """Create an asyncio.Semaphore with the configured max_concurrent."""
+        return asyncio.Semaphore(self.max_concurrent)
+
+
+# --- Alert configuration ---
+
+
+class AlertConfig(BaseModel):
+    """Alert thresholds for adjudicator monitoring.
+
+    Defines queue-depth and availability thresholds that trigger alerts
+    when the adjudicator is overloaded or unavailable.
+    """
+
+    queue_depth_warning: int = Field(
+        default=16,
+        ge=1,
+        description="Queue depth that triggers a warning alert",
+    )
+    queue_depth_critical: int = Field(
+        default=32,
+        ge=1,
+        description="Queue depth that triggers a critical alert",
+    )
+    availability_threshold_percent: float = Field(
+        default=95.0,
+        gt=0.0,
+        le=100.0,
+        description="Minimum availability percentage before alerting",
+    )
+    latency_p95_warning_ms: int = Field(
+        default=5000,
+        gt=0,
+        description="p95 latency (ms) that triggers a warning",
+    )
+    latency_p95_critical_ms: int = Field(
+        default=15000,
+        gt=0,
+        description="p95 latency (ms) that triggers a critical alert",
+    )
+    consecutive_failures_alert: int = Field(
+        default=3,
+        ge=1,
+        description="Number of consecutive failures before alerting",
+    )
+    health_check_interval_seconds: float = Field(
+        default=30.0,
+        gt=0,
+        description="Interval between health checks in seconds",
+    )

services/intelligence_pipeline_v3/adjudication/prompts.py

@@ -0,0 +1,302 @@
+"""Focused adjudication prompt building for Intelligence Pipeline v3.
+
+Builds adjudication packets containing only relevant chunks and candidates,
+uses strict JSON Schema with temperature zero, and enforces a bounded output
+budget (max 1536 tokens for decisions, not summaries).
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+from services.intelligence_pipeline_v3.adjudication.schemas import (
+    AdjudicationCandidate,
+    AdjudicationQuestion,
+    ConflictDescription,
+    EvidencePacket,
+    QuestionCode,
+)
+from services.intelligence_pipeline_v3.segmenter.models import DocumentChunk
+
+# --- Constants ---
+
+MAX_OUTPUT_TOKENS: int = 1536
+"""Maximum tokens for adjudication decisions output. Bounded to prevent
+long summaries — the adjudicator produces decisions, not narratives."""
+
+TEMPERATURE: float = 0.0
+"""Temperature for adjudication requests. Zero for deterministic output."""
+
+PROMPT_SCHEMA_VERSION: str = "1.0.0"
+"""Version of the adjudication prompt schema format."""
+
+PROVIDER_LINEAGE_KEY: str = "adjudication_v3"
+"""Lineage identifier for adjudication prompts."""
+
+
+# --- Models ---
+
+
+class PromptMetadata(BaseModel):
+    """Metadata for the adjudication prompt including version and lineage.
+
+    Tracks prompt version, schema version, and provider lineage for
+    reproducibility and auditing.
+    """
+
+    prompt_version: str = Field(
+        default="1.0.0",
+        description="Version of the prompt template",
+    )
+    schema_version: str = Field(
+        default=PROMPT_SCHEMA_VERSION,
+        description="Version of the JSON Schema format used",
+    )
+    provider_lineage: str = Field(
+        default=PROVIDER_LINEAGE_KEY,
+        description="Identifier for the prompt provider/pipeline stage",
+    )
+    max_output_tokens: int = Field(
+        default=MAX_OUTPUT_TOKENS,
+        description="Maximum output token budget for this prompt",
+    )
+    temperature: float = Field(
+        default=TEMPERATURE,
+        description="Generation temperature",
+    )
+
+
+class AdjudicationPacket(BaseModel):
+    """Complete packet sent to the 9B adjudicator.
+
+    Contains only the information relevant to resolving the specific
+    ambiguity — relevant chunks, candidates, conflicts, and questions.
+    """
+
+    document_id: str = Field(description="Source document identifier")
+    document_type: str = Field(description="Type of document")
+    relevant_chunks: list[DocumentChunk] = Field(
+        description="Only chunks relevant to the adjudication questions",
+    )
+    candidates: list[AdjudicationCandidate] = Field(
+        description="Candidates requiring adjudication",
+    )
+    conflicts: list[ConflictDescription] = Field(
+        default_factory=list,
+        description="Conflicts between candidates",
+    )
+    questions: list[AdjudicationQuestion] = Field(
+        description="Specific questions the adjudicator must answer",
+    )
+    evidence: list[EvidencePacket] = Field(
+        description="Evidence spans available for reference",
+    )
+    metadata: PromptMetadata = Field(
+        default_factory=PromptMetadata,
+        description="Prompt metadata for versioning and lineage",
+    )
+
+
+# --- Output schema for strict JSON mode ---
+
+
+def get_decision_json_schema() -> dict[str, Any]:
+    """Return the strict JSON Schema for adjudication decisions.
+
+    Used as the `response_format.json_schema.schema` payload when
+    calling the 9B model with strict structured output.
+    """
+    return {
+        "type": "object",
+        "properties": {
+            "decisions": {
+                "type": "array",
+                "items": {
+                    "type": "object",
+                    "properties": {
+                        "decision_id": {"type": "string"},
+                        "question_code": {
+                            "type": "string",
+                            "enum": [code.value for code in QuestionCode],
+                        },
+                        "verdict": {
+                            "type": "string",
+                            "enum": [
+                                "accept",
+                                "reject",
+                                "merge",
+                                "split",
+                                "reattribute",
+                            ],
+                        },
+                        "candidate_ids": {
+                            "type": "array",
+                            "items": {"type": "string"},
+                            "minItems": 1,
+                        },
+                        "evidence_ids": {
+                            "type": "array",
+                            "items": {"type": "string"},
+                            "minItems": 1,
+                        },
+                        "reasoning": {"type": "string"},
+                        "resolved_value": {"type": "object"},
+                    },
+                    "required": [
+                        "decision_id",
+                        "question_code",
+                        "verdict",
+                        "candidate_ids",
+                        "evidence_ids",
+                        "reasoning",
+                    ],
+                    "additionalProperties": False,
+                },
+            },
+        },
+        "required": ["decisions"],
+        "additionalProperties": False,
+    }
+
+
+# --- Packet builder ---
+
+
+def _get_relevant_chunk_ids(
+    candidates: list[AdjudicationCandidate],
+    conflicts: list[ConflictDescription],
+    questions: list[AdjudicationQuestion],
+) -> set[str]:
+    """Collect chunk IDs referenced by candidates, conflicts, and questions."""
+    chunk_ids: set[str] = set()
+    for candidate in candidates:
+        chunk_ids.update(candidate.source_chunk_ids)
+    return chunk_ids
+
+
+def _filter_relevant_chunks(
+    document_chunks: list[DocumentChunk],
+    relevant_chunk_ids: set[str],
+) -> list[DocumentChunk]:
+    """Filter document chunks to include only those referenced by candidates."""
+    if not relevant_chunk_ids:
+        # If no specific chunks referenced, include all (fallback for
+        # cases where chunk IDs weren't specified in candidates)
+        return document_chunks
+    return [c for c in document_chunks if c.chunk_id in relevant_chunk_ids]
+
+
+def _collect_evidence_ids(
+    candidates: list[AdjudicationCandidate],
+    conflicts: list[ConflictDescription],
+) -> set[str]:
+    """Collect all evidence IDs referenced by candidates and conflicts."""
+    evidence_ids: set[str] = set()
+    for candidate in candidates:
+        evidence_ids.update(candidate.evidence_ids)
+    for conflict in conflicts:
+        evidence_ids.update(conflict.evidence_ids)
+    return evidence_ids
+
+
+def build_adjudication_packet(
+    document_id: str,
+    document_type: str,
+    document_chunks: list[DocumentChunk],
+    candidates: list[AdjudicationCandidate],
+    conflicts: list[ConflictDescription],
+    questions: list[AdjudicationQuestion],
+    evidence: list[EvidencePacket],
+    *,
+    question_codes: list[str] | None = None,
+) -> AdjudicationPacket:
+    """Build an adjudication packet with only relevant chunks and evidence.
+
+    Filters document_chunks to include only those referenced by the
+    candidates being adjudicated. Ensures the packet is focused and
+    within the bounded context the adjudicator expects.
+
+    Args:
+        document_id: Source document identifier.
+        document_type: Type of document (article, filing, transcript, etc.).
+        document_chunks: All available chunks for the document.
+        candidates: Candidates requiring adjudication.
+        conflicts: Conflicts between candidates.
+        questions: Specific questions to resolve.
+        evidence: Available evidence spans.
+        question_codes: Optional filter to limit questions by code.
+
+    Returns:
+        AdjudicationPacket with only relevant chunks included.
+    """
+    # Filter questions by code if specified
+    filtered_questions = questions
+    if question_codes:
+        code_set = set(question_codes)
+        filtered_questions = [
+            q for q in questions if q.question_code.value in code_set
+        ]
+
+    # Determine which chunks are relevant
+    relevant_chunk_ids = _get_relevant_chunk_ids(
+        candidates, conflicts, filtered_questions
+    )
+    relevant_chunks = _filter_relevant_chunks(document_chunks, relevant_chunk_ids)
+
+    # Filter evidence to only include those referenced by candidates/conflicts
+    referenced_evidence_ids = _collect_evidence_ids(candidates, conflicts)
+    if referenced_evidence_ids:
+        relevant_evidence = [
+            e for e in evidence if e.evidence_id in referenced_evidence_ids
+        ]
+    else:
+        # Include all evidence if none specifically referenced
+        relevant_evidence = evidence
+
+    return AdjudicationPacket(
+        document_id=document_id,
+        document_type=document_type,
+        relevant_chunks=relevant_chunks,
+        candidates=candidates,
+        conflicts=conflicts,
+        questions=filtered_questions,
+        evidence=relevant_evidence,
+        metadata=PromptMetadata(),
+    )
+
+
+def build_request_payload(packet: AdjudicationPacket) -> dict[str, Any]:
+    """Build the full inference request payload for the adjudicator.
+
+    Returns a dict suitable for passing to the inference gateway,
+    including strict JSON Schema response format and temperature zero.
+    """
+    system_prompt = (
+        "You are a semantic adjudicator for financial document extraction. "
+        "Resolve the ambiguities described in the questions using ONLY the "
+        "provided evidence spans. Every decision MUST reference evidence_ids "
+        "from the provided evidence. Do NOT estimate confidence, novelty, "
+        "impact magnitude, or time horizon — those are computed by separate "
+        "calibrated pipelines. Output valid JSON matching the required schema."
+    )
+
+    user_content = packet.model_dump_json()
+
+    return {
+        "messages": [
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": user_content},
+        ],
+        "temperature": packet.metadata.temperature,
+        "max_tokens": packet.metadata.max_output_tokens,
+        "response_format": {
+            "type": "json_schema",
+            "json_schema": {
+                "name": "adjudication_response",
+                "strict": True,
+                "schema": get_decision_json_schema(),
+            },
+        },
+    }

services/intelligence_pipeline_v3/adjudication/schemas.py

@@ -0,0 +1,178 @@
+"""Adjudication schemas for Intelligence Pipeline v3.
+
+Defines Pydantic models for the adjudication layer:
+- AdjudicationCandidate: a proposed entity/fact/event requiring adjudication
+- ConflictDescription: describes a conflict between candidates
+- AdjudicationQuestion: a specific question the adjudicator must resolve
+- EvidencePacket: evidence spans provided to the adjudicator
+- AdjudicationDecision: the adjudicator's resolution (excludes confidence,
+  novelty, impact, and horizon — those come from calibrated pipelines)
+
+Every decision requires evidence_ids linking back to packet evidence.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+
+class CandidateType(str, Enum):
+    """Type of candidate being adjudicated."""
+
+    ENTITY = "entity"
+    EVENT = "event"
+    FACT = "fact"
+    RELATION = "relation"
+    SENTIMENT = "sentiment"
+
+
+class AdjudicationCandidate(BaseModel):
+    """A proposed extraction candidate that requires adjudication.
+
+    Represents an entity, event, fact, relation, or sentiment that the
+    fast-path could not resolve with sufficient confidence.
+    """
+
+    candidate_id: str = Field(description="Unique identifier for this candidate")
+    candidate_type: CandidateType = Field(description="Type of candidate")
+    label: str = Field(description="Human-readable label or description")
+    source_chunk_ids: list[str] = Field(
+        default_factory=list,
+        description="Chunk IDs where this candidate was found",
+    )
+    evidence_ids: list[str] = Field(
+        default_factory=list,
+        description="Evidence span IDs supporting this candidate",
+    )
+    metadata: dict[str, Any] = Field(
+        default_factory=dict,
+        description="Additional type-specific metadata",
+    )
+    score: float = Field(
+        default=0.0,
+        ge=0.0,
+        le=1.0,
+        description="Specialist extraction score (0.0-1.0)",
+    )
+
+
+class ConflictType(str, Enum):
+    """Type of conflict between candidates."""
+
+    CONTRADICTORY_VALUES = "contradictory_values"
+    AMBIGUOUS_IDENTITY = "ambiguous_identity"
+    OPPOSING_SENTIMENT = "opposing_sentiment"
+    OVERLAPPING_EVENTS = "overlapping_events"
+    CAUSAL_AMBIGUITY = "causal_ambiguity"
+
+
+class ConflictDescription(BaseModel):
+    """Describes a conflict between two or more candidates.
+
+    Used to inform the adjudicator about what needs resolution.
+    """
+
+    conflict_id: str = Field(description="Unique identifier for this conflict")
+    conflict_type: ConflictType = Field(description="Type of conflict")
+    candidate_ids: list[str] = Field(
+        min_length=2,
+        description="IDs of conflicting candidates",
+    )
+    description: str = Field(description="Human-readable conflict description")
+    evidence_ids: list[str] = Field(
+        default_factory=list,
+        description="Evidence IDs relevant to this conflict",
+    )
+
+
+class QuestionCode(str, Enum):
+    """Codes representing specific adjudication questions."""
+
+    RESOLVE_ENTITY_IDENTITY = "RESOLVE_ENTITY_IDENTITY"
+    RESOLVE_EVENT_TYPE = "RESOLVE_EVENT_TYPE"
+    RESOLVE_CAUSAL_DIRECTION = "RESOLVE_CAUSAL_DIRECTION"
+    RESOLVE_NUMERIC_CONFLICT = "RESOLVE_NUMERIC_CONFLICT"
+    RESOLVE_SENTIMENT_DIRECTION = "RESOLVE_SENTIMENT_DIRECTION"
+    RESOLVE_TEMPORAL_ORDERING = "RESOLVE_TEMPORAL_ORDERING"
+    RESOLVE_COMPANY_ATTRIBUTION = "RESOLVE_COMPANY_ATTRIBUTION"
+    CONFIRM_CROSS_CHUNK_RELATION = "CONFIRM_CROSS_CHUNK_RELATION"
+
+
+class AdjudicationQuestion(BaseModel):
+    """A specific question the adjudicator must answer.
+
+    Each question references candidates and conflicts that need resolution.
+    """
+
+    question_code: QuestionCode = Field(description="Structured question code")
+    description: str = Field(description="Natural language question for the adjudicator")
+    candidate_ids: list[str] = Field(
+        default_factory=list,
+        description="Candidate IDs this question applies to",
+    )
+    conflict_ids: list[str] = Field(
+        default_factory=list,
+        description="Conflict IDs this question resolves",
+    )
+
+
+class EvidencePacket(BaseModel):
+    """Evidence spans provided to the adjudicator.
+
+    Contains the exact text and location of evidence the adjudicator
+    can reference in its decisions.
+    """
+
+    evidence_id: str = Field(description="Unique identifier for this evidence span")
+    chunk_id: str = Field(description="Source chunk identifier")
+    start_char: int = Field(ge=0, description="Start character offset within chunk")
+    end_char: int = Field(gt=0, description="End character offset within chunk")
+    text: str = Field(min_length=1, description="Evidence text content")
+    source_document_id: str = Field(description="Parent document identifier")
+
+
+class DecisionVerdict(str, Enum):
+    """Possible verdicts for an adjudication decision."""
+
+    ACCEPT = "accept"
+    REJECT = "reject"
+    MERGE = "merge"
+    SPLIT = "split"
+    REATTRIBUTE = "reattribute"
+
+
+class AdjudicationDecision(BaseModel):
+    """The adjudicator's resolution for one or more candidates.
+
+    IMPORTANT: This model intentionally EXCLUDES:
+    - authoritative confidence (comes from calibration pipeline)
+    - novelty (comes from retrieval-based novelty stage)
+    - impact (comes from stock-specific impact model)
+    - horizon (comes from impact model)
+
+    The adjudicator resolves candidate identity, relationships, event
+    interpretation, and supported qualitative direction only. Every
+    decision MUST reference evidence_ids from the provided packet.
+    """
+
+    decision_id: str = Field(description="Unique identifier for this decision")
+    question_code: QuestionCode = Field(description="Which question this resolves")
+    verdict: DecisionVerdict = Field(description="The adjudication verdict")
+    candidate_ids: list[str] = Field(
+        min_length=1,
+        description="Candidate IDs this decision applies to",
+    )
+    evidence_ids: list[str] = Field(
+        min_length=1,
+        description="Evidence IDs supporting this decision (required, non-empty)",
+    )
+    reasoning: str = Field(
+        description="Brief reasoning for the decision",
+    )
+    resolved_value: dict[str, Any] = Field(
+        default_factory=dict,
+        description="The resolved value(s) if applicable",
+    )

services/intelligence_pipeline_v3/adjudication/verification.py

@@ -0,0 +1,241 @@
+"""Post-adjudication verification for Intelligence Pipeline v3.
+
+Ensures adjudication decisions are grounded in evidence, schema-compatible,
+and that repeated failures route to human review rather than accepting
+repaired defaults.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from enum import Enum
+
+from pydantic import BaseModel, Field
+
+from services.intelligence_pipeline_v3.adjudication.prompts import (
+    AdjudicationPacket,
+)
+from services.intelligence_pipeline_v3.adjudication.schemas import (
+    AdjudicationCandidate,
+    AdjudicationDecision,
+    DecisionVerdict,
+    QuestionCode,
+)
+
+# --- Models ---
+
+
+class RejectionReason(str, Enum):
+    """Reasons a decision can be rejected post-adjudication."""
+
+    MISSING_EVIDENCE_REFERENCE = "missing_evidence_reference"
+    INVALID_CANDIDATE_REFERENCE = "invalid_candidate_reference"
+    SCHEMA_INCOMPATIBLE = "schema_incompatible"
+    EMPTY_EVIDENCE_IDS = "empty_evidence_ids"
+    UNKNOWN_QUESTION_CODE = "unknown_question_code"
+    UNKNOWN_VERDICT = "unknown_verdict"
+    MISSING_REQUIRED_FIELD = "missing_required_field"
+
+
+class RejectionResult(BaseModel):
+    """Result of rejecting an unsupported or schema-incompatible decision."""
+
+    rejected: bool = Field(description="Whether the decision was rejected")
+    reasons: list[RejectionReason] = Field(
+        default_factory=list,
+        description="Reasons for rejection",
+    )
+    decision_id: str = Field(default="", description="ID of the rejected decision")
+    details: list[str] = Field(
+        default_factory=list,
+        description="Human-readable details about each rejection reason",
+    )
+
+
+class AdjudicationRecord(BaseModel):
+    """Preserves both pre-adjudication candidates and post-adjudication decisions.
+
+    This provides full audit trail showing what the pipeline proposed
+    before adjudication and what the adjudicator decided.
+    """
+
+    document_id: str = Field(description="Source document identifier")
+    pre_candidates: list[AdjudicationCandidate] = Field(
+        description="Candidates before adjudication",
+    )
+    post_decisions: list[AdjudicationDecision] = Field(
+        description="Decisions after adjudication",
+    )
+    timestamp: datetime = Field(
+        default_factory=lambda: datetime.now(timezone.utc),
+        description="When the adjudication completed",
+    )
+    packet_evidence_ids: list[str] = Field(
+        default_factory=list,
+        description="All evidence IDs that were in the adjudication packet",
+    )
+
+
+class FailureRoute(str, Enum):
+    """Possible routes for repeated failures."""
+
+    REVIEW = "review"
+    ACCEPT_REPAIRED = "accept_repaired"
+
+
+# --- Functions ---
+
+
+def verify_evidence_references(
+    decision: AdjudicationDecision,
+    packet: AdjudicationPacket,
+) -> list[str]:
+    """Check that all evidence IDs in the decision were present in the packet.
+
+    Returns a list of evidence IDs that are referenced by the decision
+    but were NOT included in the adjudication packet. An empty list
+    means all references are valid.
+
+    Args:
+        decision: The adjudication decision to verify.
+        packet: The adjudication packet that was sent to the model.
+
+    Returns:
+        List of evidence IDs that are missing from the packet (invalid refs).
+    """
+    packet_evidence_ids = {e.evidence_id for e in packet.evidence}
+    missing: list[str] = []
+    for eid in decision.evidence_ids:
+        if eid not in packet_evidence_ids:
+            missing.append(eid)
+    return missing
+
+
+def reject_unsupported_decisions(
+    decision: AdjudicationDecision,
+    *,
+    valid_candidate_ids: set[str] | None = None,
+    valid_evidence_ids: set[str] | None = None,
+) -> RejectionResult:
+    """Reject schema-incompatible or unsupported decisions.
+
+    Checks for:
+    - Empty evidence_ids (every decision must cite evidence)
+    - Invalid question codes
+    - Invalid verdicts
+    - References to non-existent candidates
+    - References to non-existent evidence (if valid sets provided)
+
+    Args:
+        decision: The decision to validate.
+        valid_candidate_ids: Optional set of valid candidate IDs.
+        valid_evidence_ids: Optional set of valid evidence IDs from the packet.
+
+    Returns:
+        RejectionResult indicating whether and why the decision was rejected.
+    """
+    reasons: list[RejectionReason] = []
+    details: list[str] = []
+
+    # Check evidence_ids is non-empty
+    if not decision.evidence_ids:
+        reasons.append(RejectionReason.EMPTY_EVIDENCE_IDS)
+        details.append("Decision has no evidence_ids — every decision must cite evidence")
+
+    # Check question_code validity
+    try:
+        QuestionCode(decision.question_code)
+    except ValueError:
+        reasons.append(RejectionReason.UNKNOWN_QUESTION_CODE)
+        details.append(f"Unknown question_code: {decision.question_code}")
+
+    # Check verdict validity
+    try:
+        DecisionVerdict(decision.verdict)
+    except ValueError:
+        reasons.append(RejectionReason.UNKNOWN_VERDICT)
+        details.append(f"Unknown verdict: {decision.verdict}")
+
+    # Check candidate references if valid set provided
+    if valid_candidate_ids is not None:
+        for cid in decision.candidate_ids:
+            if cid not in valid_candidate_ids:
+                reasons.append(RejectionReason.INVALID_CANDIDATE_REFERENCE)
+                details.append(f"Candidate ID '{cid}' not in valid set")
+                break  # One invalid ref is enough to reject
+
+    # Check evidence references if valid set provided
+    if valid_evidence_ids is not None:
+        for eid in decision.evidence_ids:
+            if eid not in valid_evidence_ids:
+                reasons.append(RejectionReason.MISSING_EVIDENCE_REFERENCE)
+                details.append(f"Evidence ID '{eid}' not in valid set")
+                break  # One invalid ref is enough to reject
+
+    # Check required fields
+    if not decision.decision_id:
+        reasons.append(RejectionReason.MISSING_REQUIRED_FIELD)
+        details.append("decision_id is empty")
+
+    if not decision.candidate_ids:
+        reasons.append(RejectionReason.MISSING_REQUIRED_FIELD)
+        details.append("candidate_ids is empty")
+
+    return RejectionResult(
+        rejected=len(reasons) > 0,
+        reasons=reasons,
+        decision_id=decision.decision_id,
+        details=details,
+    )
+
+
+def preserve_pre_and_post(
+    document_id: str,
+    pre_candidates: list[AdjudicationCandidate],
+    post_decisions: list[AdjudicationDecision],
+    packet_evidence_ids: list[str] | None = None,
+) -> AdjudicationRecord:
+    """Store both pre-adjudication candidates and final decisions.
+
+    Creates an immutable audit record preserving the full adjudication
+    state for later review and quality assessment.
+
+    Args:
+        document_id: Source document identifier.
+        pre_candidates: Candidates before adjudication.
+        post_decisions: Decisions after adjudication.
+        packet_evidence_ids: All evidence IDs from the packet.
+
+    Returns:
+        AdjudicationRecord with both pre and post states.
+    """
+    return AdjudicationRecord(
+        document_id=document_id,
+        pre_candidates=pre_candidates,
+        post_decisions=post_decisions,
+        packet_evidence_ids=packet_evidence_ids or [],
+    )
+
+
+def route_repeated_failures(failure_count: int, threshold: int) -> str:
+    """Route repeated adjudication failures to review.
+
+    When the failure count meets or exceeds the threshold, routes to
+    human review rather than accepting a repaired default. This prevents
+    the system from silently accepting potentially incorrect outputs
+    after repeated model failures.
+
+    Args:
+        failure_count: Number of consecutive adjudication failures.
+        threshold: Failure count at which to escalate to review.
+
+    Returns:
+        "review" when threshold is met/exceeded, "review" always —
+        never returns "accept_repaired" because accepting repaired
+        defaults on repeated failures undermines evidence grounding.
+    """
+    if failure_count >= threshold:
+        return FailureRoute.REVIEW.value
+    # Even below threshold, route to review for safety.
+    # The adjudication system should never silently accept repaired defaults.
+    return FailureRoute.REVIEW.value

services/intelligence_pipeline_v3/audit/__init__.py

@@ -0,0 +1,23 @@
+"""Audit and review module for the v3 intelligence pipeline.
+
+Provides evidence display, reviewer corrections, filtering by confidence/
+claims/adjudication, and immutable correction event storage.
+"""
+
+from services.intelligence_pipeline_v3.audit.models import (
+    AuditRecord,
+    CorrectionEvent,
+    CorrectionType,
+    ReviewFilter,
+    ReviewStatus,
+)
+from services.intelligence_pipeline_v3.audit.store import AuditStore
+
+__all__ = [
+    "AuditRecord",
+    "AuditStore",
+    "CorrectionEvent",
+    "CorrectionType",
+    "ReviewFilter",
+    "ReviewStatus",
+]

services/intelligence_pipeline_v3/audit/models.py

@@ -0,0 +1,201 @@
+"""Audit and review data models.
+
+Supports evidence display with offsets, specialist probabilities,
+routing reasons, adjudicator decisions, impact-model outputs,
+and immutable reviewer correction events.
+"""
+
+from __future__ import annotations
+
+import enum
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID, uuid4
+
+
+class ReviewStatus(str, enum.Enum):
+    """Status of a document's review."""
+
+    PENDING = "pending"
+    REVIEWED = "reviewed"
+    CORRECTED = "corrected"
+    CONFIRMED = "confirmed"
+
+
+class CorrectionType(str, enum.Enum):
+    """Types of reviewer corrections."""
+
+    CORRECT = "correct"
+    INCORRECT = "incorrect"
+    UNSUPPORTED = "unsupported"
+    AMBIGUOUS = "ambiguous"
+    VALUE_OVERRIDE = "value_override"
+
+
+@dataclass(frozen=True)
+class CorrectionEvent:
+    """Immutable reviewer correction event.
+
+    Corrections are append-only audit events. They feed the active-learning
+    dataset only through the approved export process.
+    """
+
+    event_id: UUID
+    record_id: UUID
+    field_name: str
+    correction_type: CorrectionType
+    original_value: Any
+    corrected_value: Any | None
+    reviewer_id: str
+    timestamp: datetime
+    notes: str = ""
+
+    @classmethod
+    def create(
+        cls,
+        record_id: UUID,
+        field_name: str,
+        correction_type: CorrectionType,
+        original_value: Any,
+        corrected_value: Any | None = None,
+        reviewer_id: str = "",
+        notes: str = "",
+    ) -> CorrectionEvent:
+        return cls(
+            event_id=uuid4(),
+            record_id=record_id,
+            field_name=field_name,
+            correction_type=correction_type,
+            original_value=original_value,
+            corrected_value=corrected_value,
+            reviewer_id=reviewer_id,
+            timestamp=datetime.now(timezone.utc),
+            notes=notes,
+        )
+
+
+@dataclass
+class AuditRecord:
+    """Complete audit record for a processed document.
+
+    Contains source evidence, specialist outputs, routing reasons,
+    adjudicator decisions, and impact predictions displayed separately.
+    """
+
+    record_id: UUID
+    document_id: str
+    run_id: UUID
+    timestamp: datetime
+
+    # Source evidence with offsets
+    evidence_spans: list[dict[str, Any]] = field(default_factory=list)
+
+    # Specialist stage outputs (probabilities, scores)
+    specialist_outputs: dict[str, Any] = field(default_factory=dict)
+
+    # Routing decision and reasons
+    routing_reasons: list[str] = field(default_factory=list)
+    route_decision: str = ""
+
+    # Adjudicator decision (if applicable)
+    adjudicator_decision: dict[str, Any] | None = None
+
+    # Impact model outputs
+    impact_outputs: dict[str, Any] = field(default_factory=dict)
+
+    # Model lineage
+    lineage: dict[str, Any] = field(default_factory=dict)
+
+    # Review status
+    review_status: ReviewStatus = ReviewStatus.PENDING
+    corrections: list[CorrectionEvent] = field(default_factory=list)
+
+    @classmethod
+    def create(
+        cls,
+        document_id: str,
+        run_id: UUID,
+        evidence_spans: list[dict[str, Any]] | None = None,
+        specialist_outputs: dict[str, Any] | None = None,
+        routing_reasons: list[str] | None = None,
+        route_decision: str = "",
+        adjudicator_decision: dict[str, Any] | None = None,
+        impact_outputs: dict[str, Any] | None = None,
+        lineage: dict[str, Any] | None = None,
+    ) -> AuditRecord:
+        return cls(
+            record_id=uuid4(),
+            document_id=document_id,
+            run_id=run_id,
+            timestamp=datetime.now(timezone.utc),
+            evidence_spans=evidence_spans or [],
+            specialist_outputs=specialist_outputs or {},
+            routing_reasons=routing_reasons or [],
+            route_decision=route_decision,
+            adjudicator_decision=adjudicator_decision,
+            impact_outputs=impact_outputs or {},
+            lineage=lineage or {},
+        )
+
+    def add_correction(self, correction: CorrectionEvent) -> None:
+        """Add an immutable correction event."""
+        self.corrections.append(correction)
+        self.review_status = ReviewStatus.CORRECTED
+
+    def mark_reviewed(self) -> None:
+        """Mark the record as reviewed without corrections."""
+        if self.review_status == ReviewStatus.PENDING:
+            self.review_status = ReviewStatus.REVIEWED
+
+    def mark_confirmed(self) -> None:
+        """Mark the record as confirmed correct."""
+        self.review_status = ReviewStatus.CONFIRMED
+
+
+@dataclass
+class ReviewFilter:
+    """Filter criteria for audit records.
+
+    Supports filtering by confidence, unsupported claims, adjudication
+    status, review status, and date ranges.
+    """
+
+    min_confidence: float | None = None
+    max_confidence: float | None = None
+    has_unsupported_claims: bool | None = None
+    is_adjudicated: bool | None = None
+    review_status: ReviewStatus | None = None
+    document_type: str | None = None
+    company_id: UUID | None = None
+    from_date: datetime | None = None
+    to_date: datetime | None = None
+
+    def matches(self, record: AuditRecord) -> bool:
+        """Check if a record matches this filter."""
+        if self.is_adjudicated is not None:
+            has_adj = record.adjudicator_decision is not None
+            if has_adj != self.is_adjudicated:
+                return False
+
+        if self.review_status is not None:
+            if record.review_status != self.review_status:
+                return False
+
+        if self.from_date is not None:
+            if record.timestamp < self.from_date:
+                return False
+
+        if self.to_date is not None:
+            if record.timestamp > self.to_date:
+                return False
+
+        if self.has_unsupported_claims is not None:
+            has_unsupported = any(
+                c.correction_type == CorrectionType.UNSUPPORTED
+                for c in record.corrections
+            )
+            if has_unsupported != self.has_unsupported_claims:
+                return False
+
+        return True

services/intelligence_pipeline_v3/audit/store.py

@@ -0,0 +1,81 @@
+"""Audit record storage with filtering and retrieval.
+
+In production, this would be backed by PostgreSQL.
+This implementation provides the storage interface for testing.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from uuid import UUID
+
+from services.intelligence_pipeline_v3.audit.models import (
+    AuditRecord,
+    CorrectionEvent,
+    ReviewFilter,
+)
+
+
+@dataclass
+class AuditStore:
+    """In-memory audit record store with filtering.
+
+    Provides storage, retrieval, and filtering of audit records and
+    their immutable correction events.
+    """
+
+    _records: dict[UUID, AuditRecord] = field(default_factory=dict)
+    _corrections: list[CorrectionEvent] = field(default_factory=list)
+
+    def store(self, record: AuditRecord) -> None:
+        """Store an audit record."""
+        self._records[record.record_id] = record
+
+    def get(self, record_id: UUID) -> AuditRecord | None:
+        """Retrieve a record by ID."""
+        return self._records.get(record_id)
+
+    def get_by_document(self, document_id: str) -> list[AuditRecord]:
+        """Get all records for a document."""
+        return [
+            r for r in self._records.values() if r.document_id == document_id
+        ]
+
+    def get_by_run(self, run_id: UUID) -> AuditRecord | None:
+        """Get the record for a pipeline run."""
+        for r in self._records.values():
+            if r.run_id == run_id:
+                return r
+        return None
+
+    def add_correction(
+        self, record_id: UUID, correction: CorrectionEvent
+    ) -> bool:
+        """Add a correction to a record. Returns False if record not found."""
+        record = self._records.get(record_id)
+        if record is None:
+            return False
+        record.add_correction(correction)
+        self._corrections.append(correction)
+        return True
+
+    def filter(self, criteria: ReviewFilter) -> list[AuditRecord]:
+        """Filter records by criteria."""
+        return [
+            r for r in self._records.values() if criteria.matches(r)
+        ]
+
+    def get_corrections(self, record_id: UUID) -> list[CorrectionEvent]:
+        """Get all corrections for a record."""
+        record = self._records.get(record_id)
+        if record is None:
+            return []
+        return list(record.corrections)
+
+    def count(self) -> int:
+        """Total stored records."""
+        return len(self._records)
+
+    def correction_count(self) -> int:
+        """Total correction events across all records."""
+        return len(self._corrections)

services/intelligence_pipeline_v3/benchmark/__init__.py

@@ -0,0 +1,46 @@
+"""Benchmark configuration and comparison framework for Intelligence Pipeline v3.
+
+Defines extraction configurations for controlled comparison between the current
+production pipeline and corrected variants. Supports attribution of improvement
+sources (temperature fix, schema constraints, architecture changes).
+
+Validates: Requirements 16.2, 16.3, 16.5
+"""
+
+from services.intelligence_pipeline_v3.benchmark.comparison import (
+    ComparisonReport,
+    ConfigDelta,
+    FieldDelta,
+    ResourceDelta,
+    compare_configurations,
+)
+from services.intelligence_pipeline_v3.benchmark.configurations import (
+    BASELINE_CURRENT,
+    BASELINE_STRICT_SCHEMA,
+    BASELINE_TEMP_ZERO,
+    BenchmarkConfig,
+    StructuredOutputMode,
+    list_configurations,
+)
+from services.intelligence_pipeline_v3.benchmark.runner import (
+    BenchmarkDocumentResult,
+    BenchmarkRun,
+    BenchmarkRunner,
+)
+
+__all__ = [
+    "BASELINE_CURRENT",
+    "BASELINE_STRICT_SCHEMA",
+    "BASELINE_TEMP_ZERO",
+    "BenchmarkConfig",
+    "BenchmarkDocumentResult",
+    "BenchmarkRun",
+    "BenchmarkRunner",
+    "ComparisonReport",
+    "ConfigDelta",
+    "FieldDelta",
+    "ResourceDelta",
+    "StructuredOutputMode",
+    "compare_configurations",
+    "list_configurations",
+]

services/intelligence_pipeline_v3/benchmark/comparison.py

@@ -0,0 +1,294 @@
+"""Comparison and attribution for benchmark configurations.
+
+Produces delta tables and attribution reports to quantify how much of
+the apparent architecture gain comes from fixing the current request alone
+(temperature, schema constraints) versus the full v3 architecture.
+
+Validates: Requirements 16.2, 16.3, 16.5
+"""
+from __future__ import annotations
+
+from pydantic import BaseModel, Field
+
+from services.intelligence_pipeline_v3.benchmark.runner import BenchmarkRun
+
+# ---------------------------------------------------------------------------
+# Result Models
+# ---------------------------------------------------------------------------
+
+
+class FieldDelta(BaseModel):
+    """Per-field improvement between two configurations."""
+
+    field_name: str = Field(description="Name of the compared field/metric")
+    baseline_value: float = Field(description="Value in the baseline configuration")
+    comparison_value: float = Field(description="Value in the compared configuration")
+    absolute_delta: float = Field(description="comparison - baseline")
+    relative_delta_percent: float = Field(
+        description="Percentage change from baseline ((comp - base) / base * 100)",
+    )
+    improved: bool = Field(
+        description="Whether the delta represents improvement (higher is better assumed unless inverted)",
+    )
+
+
+class ResourceDelta(BaseModel):
+    """Resource usage comparison between configurations."""
+
+    metric_name: str = Field(description="Resource metric name")
+    baseline_value: float = Field(description="Baseline resource usage")
+    comparison_value: float = Field(description="Compared configuration resource usage")
+    absolute_delta: float = Field(description="comparison - baseline")
+    relative_delta_percent: float = Field(description="Percentage change")
+    improved: bool = Field(
+        description="Whether the delta represents improvement (lower is better for resources)",
+    )
+
+
+class ConfigDelta(BaseModel):
+    """Comparison results between a baseline and one other configuration."""
+
+    baseline_config: str = Field(description="Baseline configuration name")
+    comparison_config: str = Field(description="Compared configuration name")
+    field_deltas: list[FieldDelta] = Field(default_factory=list)
+    resource_deltas: list[ResourceDelta] = Field(default_factory=list)
+
+
+class ComparisonReport(BaseModel):
+    """Full comparison report across multiple configurations.
+
+    Attributes:
+        configs_compared: Names of all configurations in this comparison.
+        deltas: Per-configuration comparison against the baseline.
+        attribution_summary: Human-readable attribution of improvement sources.
+    """
+
+    configs_compared: list[str] = Field(
+        description="All configuration names included in this comparison",
+    )
+    deltas: list[ConfigDelta] = Field(
+        default_factory=list,
+        description="Delta tables for each non-baseline config vs baseline",
+    )
+    attribution_summary: dict[str, float] = Field(
+        default_factory=dict,
+        description=(
+            "Attribution percentages: maps source (e.g. 'temperature_fix', "
+            "'schema_constraint', 'architecture') to fraction of total improvement"
+        ),
+    )
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _compute_field_delta(
+    field_name: str,
+    baseline_val: float,
+    comparison_val: float,
+    *,
+    higher_is_better: bool = True,
+) -> FieldDelta:
+    """Compute a single field delta with direction awareness."""
+    absolute = comparison_val - baseline_val
+    relative = (
+        (absolute / baseline_val * 100.0) if baseline_val != 0.0 else 0.0
+    )
+    improved = absolute > 0.0 if higher_is_better else absolute < 0.0
+
+    return FieldDelta(
+        field_name=field_name,
+        baseline_value=baseline_val,
+        comparison_value=comparison_val,
+        absolute_delta=absolute,
+        relative_delta_percent=relative,
+        improved=improved,
+    )
+
+
+def _compute_resource_delta(
+    metric_name: str,
+    baseline_val: float,
+    comparison_val: float,
+) -> ResourceDelta:
+    """Compute a resource delta (lower is better)."""
+    absolute = comparison_val - baseline_val
+    relative = (
+        (absolute / baseline_val * 100.0) if baseline_val != 0.0 else 0.0
+    )
+    improved = absolute < 0.0  # Lower resource usage is better
+
+    return ResourceDelta(
+        metric_name=metric_name,
+        baseline_value=baseline_val,
+        comparison_value=comparison_val,
+        absolute_delta=absolute,
+        relative_delta_percent=relative,
+        improved=improved,
+    )
+
+
+def _run_metrics(run: BenchmarkRun) -> dict[str, float]:
+    """Extract summary metrics from a benchmark run."""
+    n = len(run.results) or 1  # Avoid division by zero
+
+    return {
+        "schema_validity_rate": run.schema_validity_rate,
+        "success_count": float(run.success_count),
+        "failure_count": float(run.failure_count),
+        "mean_duration_ms": run.mean_duration_ms,
+        "total_input_tokens": float(run.total_input_tokens),
+        "total_output_tokens": float(run.total_output_tokens),
+        "mean_retries": sum(r.retries for r in run.results) / n,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+
+def compare_configurations(
+    baseline_run: BenchmarkRun,
+    comparison_runs: list[BenchmarkRun],
+) -> ComparisonReport:
+    """Compare benchmark runs to produce delta tables and attribution.
+
+    Computes per-field and per-resource deltas between the baseline and
+    each comparison configuration, then attributes improvement sources.
+
+    Args:
+        baseline_run: The baseline (typically BASELINE_CURRENT) run results.
+        comparison_runs: One or more comparison configuration runs.
+
+    Returns:
+        ComparisonReport with deltas and attribution percentages.
+    """
+    configs_compared = [baseline_run.config_name] + [
+        r.config_name for r in comparison_runs
+    ]
+
+    baseline_metrics = _run_metrics(baseline_run)
+    deltas: list[ConfigDelta] = []
+
+    # Fields where higher is better
+    _higher_is_better = {"schema_validity_rate", "success_count"}
+    # Fields where lower is better (resource-like)
+    _resource_fields = {
+        "mean_duration_ms",
+        "total_input_tokens",
+        "total_output_tokens",
+        "mean_retries",
+        "failure_count",
+    }
+
+    for comp_run in comparison_runs:
+        comp_metrics = _run_metrics(comp_run)
+        field_deltas: list[FieldDelta] = []
+        resource_deltas: list[ResourceDelta] = []
+
+        for metric_name, baseline_val in baseline_metrics.items():
+            comp_val = comp_metrics[metric_name]
+
+            if metric_name in _resource_fields:
+                resource_deltas.append(
+                    _compute_resource_delta(metric_name, baseline_val, comp_val)
+                )
+            else:
+                field_deltas.append(
+                    _compute_field_delta(
+                        metric_name,
+                        baseline_val,
+                        comp_val,
+                        higher_is_better=(metric_name in _higher_is_better),
+                    )
+                )
+
+        deltas.append(
+            ConfigDelta(
+                baseline_config=baseline_run.config_name,
+                comparison_config=comp_run.config_name,
+                field_deltas=field_deltas,
+                resource_deltas=resource_deltas,
+            )
+        )
+
+    # Attribution: estimate how much improvement comes from each fix
+    attribution = _compute_attribution(baseline_metrics, comparison_runs)
+
+    return ComparisonReport(
+        configs_compared=configs_compared,
+        deltas=deltas,
+        attribution_summary=attribution,
+    )
+
+
+def _compute_attribution(
+    baseline_metrics: dict[str, float],
+    comparison_runs: list[BenchmarkRun],
+) -> dict[str, float]:
+    """Compute attribution percentages for improvement sources.
+
+    Uses schema_validity_rate as the primary improvement signal.
+    Attribution is computed as the fraction of total improvement each
+    configuration step contributes.
+
+    Returns a dict mapping source labels to fraction (0.0-1.0).
+    """
+    attribution: dict[str, float] = {}
+
+    if not comparison_runs:
+        return attribution
+
+    baseline_validity = baseline_metrics["schema_validity_rate"]
+
+    # Find temp_zero and strict_schema runs by config name
+    temp_zero_validity: float | None = None
+    strict_schema_validity: float | None = None
+
+    for run in comparison_runs:
+        run_metrics = _run_metrics(run)
+        if "temp_zero" in run.config_name:
+            temp_zero_validity = run_metrics["schema_validity_rate"]
+        elif "strict_schema" in run.config_name:
+            strict_schema_validity = run_metrics["schema_validity_rate"]
+
+    # Compute incremental gains
+    # Total improvement = strict_schema - baseline (or best comparison - baseline)
+    best_validity = max(
+        _run_metrics(r)["schema_validity_rate"] for r in comparison_runs
+    )
+    total_improvement = best_validity - baseline_validity
+
+    if total_improvement <= 0.0:
+        # No improvement detected; equal attribution
+        attribution["temperature_fix"] = 0.0
+        attribution["schema_constraint"] = 0.0
+        return attribution
+
+    # Temperature fix contribution
+    if temp_zero_validity is not None:
+        temp_gain = temp_zero_validity - baseline_validity
+        attribution["temperature_fix"] = max(0.0, temp_gain / total_improvement)
+    else:
+        attribution["temperature_fix"] = 0.0
+
+    # Schema constraint contribution (incremental over temp fix)
+    if strict_schema_validity is not None and temp_zero_validity is not None:
+        schema_gain = strict_schema_validity - temp_zero_validity
+        attribution["schema_constraint"] = max(0.0, schema_gain / total_improvement)
+    elif strict_schema_validity is not None:
+        schema_gain = strict_schema_validity - baseline_validity
+        attribution["schema_constraint"] = max(0.0, schema_gain / total_improvement)
+    else:
+        attribution["schema_constraint"] = 0.0
+
+    # Remaining is attributed to other factors
+    accounted = attribution.get("temperature_fix", 0.0) + attribution.get(
+        "schema_constraint", 0.0
+    )
+    attribution["other"] = max(0.0, 1.0 - accounted)
+
+    return attribution

services/intelligence_pipeline_v3/benchmark/configurations.py

@@ -0,0 +1,134 @@
+"""Benchmark configuration definitions for controlled extraction comparisons.
+
+Defines the standard configurations used to attribute improvement sources:
+- BASELINE_CURRENT: Current production settings (temperature 0.7, no schema constraint)
+- BASELINE_TEMP_ZERO: Same model, temperature 0, no schema constraint
+- BASELINE_STRICT_SCHEMA: Same model, temperature 0, strict JSON Schema
+
+Validates: Requirements 16.2, 16.3, 16.5
+"""
+from __future__ import annotations
+
+from enum import Enum
+from typing import Any
+
+from pydantic import BaseModel, ConfigDict, Field
+
+
+class StructuredOutputMode(str, Enum):
+    """Structured output constraint modes for extraction."""
+
+    NONE = "none"
+    JSON_OBJECT = "json_object"
+    JSON_SCHEMA = "json_schema"
+
+
+class BenchmarkConfig(BaseModel):
+    """Configuration for a single benchmark extraction run.
+
+    Captures all parameters that affect extraction behavior so that
+    differences between runs can be attributed to specific settings.
+    """
+
+    model_config = ConfigDict(frozen=True)
+
+    config_name: str = Field(
+        description="Unique identifier for this configuration",
+    )
+    description: str = Field(
+        description="Human-readable description of what this configuration tests",
+    )
+    model_name: str = Field(
+        description="Served model name (e.g. 'AxionML/Qwen3.5-9B-NVFP4')",
+    )
+    temperature: float = Field(
+        ge=0.0,
+        le=2.0,
+        description="Sampling temperature; 0.0 = deterministic",
+    )
+    max_output_tokens: int = Field(
+        gt=0,
+        description="Maximum tokens in generated output",
+    )
+    structured_output_mode: StructuredOutputMode = Field(
+        description="How output structure is constrained",
+    )
+    seed: int | None = Field(
+        default=None,
+        description="Random seed for reproducibility (None = not pinned)",
+    )
+    additional_params: dict[str, Any] = Field(
+        default_factory=dict,
+        description="Provider-specific extra parameters",
+    )
+
+
+# ---------------------------------------------------------------------------
+# Standard Benchmark Configurations
+# ---------------------------------------------------------------------------
+
+# The 9B model currently deployed on the cluster
+_DEFAULT_MODEL = "AxionML/Qwen3.5-9B-NVFP4"
+_DEFAULT_MAX_OUTPUT_TOKENS = 2048
+
+BASELINE_CURRENT = BenchmarkConfig(
+    config_name="baseline_current",
+    description=(
+        "Current production settings: temperature 0.7, response_format json_object "
+        "only (schema not enforced on generation), no seed pinning. "
+        "Represents the unchanged request as deployed."
+    ),
+    model_name=_DEFAULT_MODEL,
+    temperature=0.7,
+    max_output_tokens=_DEFAULT_MAX_OUTPUT_TOKENS,
+    structured_output_mode=StructuredOutputMode.JSON_OBJECT,
+    seed=None,
+    additional_params={},
+)
+
+BASELINE_TEMP_ZERO = BenchmarkConfig(
+    config_name="baseline_temp_zero",
+    description=(
+        "Same 9B model with temperature set to 0.0 for deterministic generation. "
+        "Still uses json_object mode without strict schema enforcement. "
+        "Isolates the effect of removing sampling stochasticity."
+    ),
+    model_name=_DEFAULT_MODEL,
+    temperature=0.0,
+    max_output_tokens=_DEFAULT_MAX_OUTPUT_TOKENS,
+    structured_output_mode=StructuredOutputMode.JSON_OBJECT,
+    seed=0,
+    additional_params={},
+)
+
+BASELINE_STRICT_SCHEMA = BenchmarkConfig(
+    config_name="baseline_strict_schema",
+    description=(
+        "Same 9B model with temperature 0.0 AND strict JSON Schema output "
+        "enforcement via vLLM structured output backend. "
+        "Isolates the combined effect of deterministic generation plus "
+        "grammar-constrained decoding."
+    ),
+    model_name=_DEFAULT_MODEL,
+    temperature=0.0,
+    max_output_tokens=_DEFAULT_MAX_OUTPUT_TOKENS,
+    structured_output_mode=StructuredOutputMode.JSON_SCHEMA,
+    seed=0,
+    additional_params={},
+)
+
+# Registry of all standard configurations
+_STANDARD_CONFIGURATIONS: dict[str, BenchmarkConfig] = {
+    BASELINE_CURRENT.config_name: BASELINE_CURRENT,
+    BASELINE_TEMP_ZERO.config_name: BASELINE_TEMP_ZERO,
+    BASELINE_STRICT_SCHEMA.config_name: BASELINE_STRICT_SCHEMA,
+}
+
+
+def list_configurations() -> list[BenchmarkConfig]:
+    """Return all registered benchmark configurations.
+
+    Returns:
+        List of BenchmarkConfig instances in definition order.
+    """
+    return list(_STANDARD_CONFIGURATIONS.values())

services/intelligence_pipeline_v3/benchmark/runner.py

@@ -0,0 +1,296 @@
+"""Benchmark runner scaffold for extraction configuration comparisons.
+
+Provides the framework for running extraction benchmarks across different
+configurations. The actual model invocations require the cluster, but
+results can be stored and compared locally.
+
+Validates: Requirements 16.2, 16.3
+"""
+from __future__ import annotations
+
+import json
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+from services.intelligence_pipeline_v3.benchmark.configurations import (
+    BenchmarkConfig,
+)
+
+# ---------------------------------------------------------------------------
+# Result Models
+# ---------------------------------------------------------------------------
+
+
+class BenchmarkDocumentResult(BaseModel):
+    """Result of running one document through one benchmark configuration."""
+
+    document_id: str = Field(description="Identifier of the source document")
+    raw_output: str | None = Field(
+        default=None,
+        description="Raw model output text (before parsing)",
+    )
+    parsed_output: dict[str, Any] | None = Field(
+        default=None,
+        description="Parsed JSON output if extraction succeeded",
+    )
+    schema_valid: bool = Field(
+        default=False,
+        description="Whether the output passed JSON Schema validation",
+    )
+    retries: int = Field(
+        default=0,
+        ge=0,
+        description="Number of retries needed to get valid output",
+    )
+    duration_ms: int = Field(
+        default=0,
+        ge=0,
+        description="Total wall-clock time in milliseconds",
+    )
+    input_tokens: int = Field(
+        default=0,
+        ge=0,
+        description="Input tokens consumed",
+    )
+    output_tokens: int = Field(
+        default=0,
+        ge=0,
+        description="Output tokens generated",
+    )
+    error: str | None = Field(
+        default=None,
+        description="Error message if extraction failed",
+    )
+
+
+class BenchmarkRun(BaseModel):
+    """A complete benchmark run: one configuration applied to multiple documents."""
+
+    config_name: str = Field(description="Configuration used for this run")
+    timestamp: datetime = Field(
+        default_factory=lambda: datetime.now(timezone.utc),
+        description="When this run was executed",
+    )
+    document_ids: list[str] = Field(
+        default_factory=list,
+        description="Documents included in this run",
+    )
+    results: list[BenchmarkDocumentResult] = Field(
+        default_factory=list,
+        description="Per-document results",
+    )
+
+    @property
+    def success_count(self) -> int:
+        """Number of documents that produced valid output."""
+        return sum(1 for r in self.results if r.schema_valid and r.error is None)
+
+    @property
+    def failure_count(self) -> int:
+        """Number of documents that failed or produced invalid output."""
+        return len(self.results) - self.success_count
+
+    @property
+    def schema_validity_rate(self) -> float:
+        """Fraction of results that passed schema validation."""
+        if not self.results:
+            return 0.0
+        return self.success_count / len(self.results)
+
+    @property
+    def mean_duration_ms(self) -> float:
+        """Average duration across all results."""
+        if not self.results:
+            return 0.0
+        return sum(r.duration_ms for r in self.results) / len(self.results)
+
+    @property
+    def total_input_tokens(self) -> int:
+        """Total input tokens across all results."""
+        return sum(r.input_tokens for r in self.results)
+
+    @property
+    def total_output_tokens(self) -> int:
+        """Total output tokens across all results."""
+        return sum(r.output_tokens for r in self.results)
+
+
+# ---------------------------------------------------------------------------
+# Artifact Storage
+# ---------------------------------------------------------------------------
+
+_DEFAULT_ARTIFACT_DIR = Path("artifacts/benchmark")
+
+
+def _ensure_artifact_dir(base: Path) -> Path:
+    """Create artifact directory if it does not exist."""
+    base.mkdir(parents=True, exist_ok=True)
+    return base
+
+
+def save_benchmark_run(
+    run: BenchmarkRun,
+    artifact_dir: Path | None = None,
+) -> Path:
+    """Persist a benchmark run as a JSON artifact.
+
+    Args:
+        run: The benchmark run to save.
+        artifact_dir: Directory to write to. Defaults to artifacts/benchmark/.
+
+    Returns:
+        Path to the written JSON file.
+    """
+    base = artifact_dir or _DEFAULT_ARTIFACT_DIR
+    _ensure_artifact_dir(base)
+
+    ts = run.timestamp.strftime("%Y%m%d_%H%M%S")
+    filename = f"{run.config_name}_{ts}.json"
+    path = base / filename
+
+    path.write_text(
+        run.model_dump_json(indent=2),
+        encoding="utf-8",
+    )
+    return path
+
+
+def load_benchmark_run(path: Path) -> BenchmarkRun:
+    """Load a benchmark run from a JSON artifact.
+
+    Args:
+        path: Path to the JSON artifact file.
+
+    Returns:
+        Deserialized BenchmarkRun.
+    """
+    data = json.loads(path.read_text(encoding="utf-8"))
+    return BenchmarkRun.model_validate(data)
+
+
+# ---------------------------------------------------------------------------
+# Runner
+# ---------------------------------------------------------------------------
+
+
+class BenchmarkRunner:
+    """Runs extraction benchmarks using a given configuration.
+
+    The runner provides the scaffolding for executing benchmarks.
+    Actual LLM invocation is delegated to an inference callable.
+    When no inference callable is provided, results are recorded as
+    errors (useful for dry-run / configuration testing).
+    """
+
+    def __init__(
+        self,
+        config: BenchmarkConfig,
+        artifact_dir: Path | None = None,
+        inference_fn: Any | None = None,
+    ) -> None:
+        """Initialize the benchmark runner.
+
+        Args:
+            config: Benchmark configuration to use for all runs.
+            artifact_dir: Where to store result artifacts.
+            inference_fn: Optional async callable(document_text, config) -> dict.
+                If None, documents are recorded as not-run errors.
+        """
+        self.config = config
+        self.artifact_dir = artifact_dir or _DEFAULT_ARTIFACT_DIR
+        self._inference_fn = inference_fn
+
+    async def run_single_document(
+        self,
+        document_id: str,
+        document_text: str,
+        json_schema: dict[str, Any] | None = None,
+    ) -> BenchmarkDocumentResult:
+        """Run a single document through the configured extraction.
+
+        Args:
+            document_id: Unique document identifier.
+            document_text: Full document text to extract from.
+            json_schema: Optional JSON Schema for validation.
+
+        Returns:
+            BenchmarkDocumentResult with extraction outcome.
+        """
+        if self._inference_fn is None:
+            return BenchmarkDocumentResult(
+                document_id=document_id,
+                error="No inference function configured (dry-run mode)",
+            )
+
+        start = time.perf_counter()
+        try:
+            result = await self._inference_fn(document_text, self.config)
+            duration_ms = int((time.perf_counter() - start) * 1000)
+
+            raw_output = result.get("raw_output", "")
+            parsed_output = result.get("parsed_output")
+            schema_valid = result.get("schema_valid", False)
+            input_tokens = result.get("input_tokens", 0)
+            output_tokens = result.get("output_tokens", 0)
+            retries = result.get("retries", 0)
+
+            return BenchmarkDocumentResult(
+                document_id=document_id,
+                raw_output=raw_output,
+                parsed_output=parsed_output,
+                schema_valid=schema_valid,
+                retries=retries,
+                duration_ms=duration_ms,
+                input_tokens=input_tokens,
+                output_tokens=output_tokens,
+            )
+        except Exception as exc:
+            duration_ms = int((time.perf_counter() - start) * 1000)
+            return BenchmarkDocumentResult(
+                document_id=document_id,
+                duration_ms=duration_ms,
+                error=str(exc),
+            )
+
+    async def run_batch(
+        self,
+        documents: list[tuple[str, str]],
+        json_schema: dict[str, Any] | None = None,
+        save_artifacts: bool = True,
+    ) -> BenchmarkRun:
+        """Run a batch of documents through the configured extraction.
+
+        Args:
+            documents: List of (document_id, document_text) tuples.
+            json_schema: Optional JSON Schema for validation.
+            save_artifacts: Whether to persist results as JSON artifacts.
+
+        Returns:
+            BenchmarkRun with all document results.
+        """
+        results: list[BenchmarkDocumentResult] = []
+        document_ids: list[str] = []
+
+        for doc_id, doc_text in documents:
+            document_ids.append(doc_id)
+            result = await self.run_single_document(
+                document_id=doc_id,
+                document_text=doc_text,
+                json_schema=json_schema,
+            )
+            results.append(result)
+
+        run = BenchmarkRun(
+            config_name=self.config.config_name,
+            document_ids=document_ids,
+            results=results,
+        )
+
+        if save_artifacts:
+            save_benchmark_run(run, self.artifact_dir)
+
+        return run

services/intelligence_pipeline_v3/canary/__init__.py

@@ -0,0 +1,28 @@
+"""Canary deployment module for v3 pipeline promotion.
+
+Supports percentage-based routing, automatic rollback on threshold
+violations, audit integrity during rollback, and paper-trading
+signal influence with divergence review.
+"""
+
+from services.intelligence_pipeline_v3.canary.influence import (
+    DivergenceRecord,
+    SignalInfluenceConfig,
+    SignalInfluenceTracker,
+)
+from services.intelligence_pipeline_v3.canary.routing import (
+    CanaryConfig,
+    CanaryRouter,
+    RollbackEvent,
+    RollbackReason,
+)
+
+__all__ = [
+    "CanaryConfig",
+    "CanaryRouter",
+    "DivergenceRecord",
+    "RollbackEvent",
+    "RollbackReason",
+    "SignalInfluenceConfig",
+    "SignalInfluenceTracker",
+]

services/intelligence_pipeline_v3/canary/influence.py

@@ -0,0 +1,187 @@
+"""Canary signal influence — paper trading with v3 signals.
+
+Enables v3 signals in paper trading at a small percentage, tracks
+extraction correctness separately from trading outcomes, reviews
+material recommendation divergences, and requires explicit owner
+approval for full promotion.
+"""
+
+from __future__ import annotations
+
+import enum
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID, uuid4
+
+
+class PromotionStatus(str, enum.Enum):
+    """Status of the canary promotion process."""
+
+    PENDING = "pending"
+    PAPER_TRADING = "paper_trading"
+    AWAITING_REVIEW = "awaiting_review"
+    APPROVED = "approved"
+    REJECTED = "rejected"
+
+
+@dataclass
+class DivergenceRecord:
+    """Record of a material recommendation divergence between v2 and v3."""
+
+    record_id: UUID
+    document_id: str
+    timestamp: datetime
+    v2_recommendation: dict[str, Any]
+    v3_recommendation: dict[str, Any]
+    divergence_type: str  # e.g., "direction_opposite", "magnitude_significant"
+    impact_estimate: float = 0.0  # Estimated impact on portfolio
+    reviewed: bool = False
+    reviewer_notes: str = ""
+
+    @classmethod
+    def create(
+        cls,
+        document_id: str,
+        v2_recommendation: dict[str, Any],
+        v3_recommendation: dict[str, Any],
+        divergence_type: str,
+        impact_estimate: float = 0.0,
+    ) -> DivergenceRecord:
+        return cls(
+            record_id=uuid4(),
+            document_id=document_id,
+            timestamp=datetime.now(timezone.utc),
+            v2_recommendation=v2_recommendation,
+            v3_recommendation=v3_recommendation,
+            divergence_type=divergence_type,
+            impact_estimate=impact_estimate,
+        )
+
+
+@dataclass
+class SignalInfluenceConfig:
+    """Configuration for canary signal influence in paper trading."""
+
+    enabled: bool = False
+    percentage: int = 5  # Start at 5% of paper trading signals
+    require_owner_approval: bool = True
+    owner_id: str = ""
+
+    # Reporting thresholds
+    material_divergence_threshold: float = 0.20
+    max_divergence_rate: float = 0.15
+
+    # Separation of concerns
+    report_extraction_separately: bool = True
+    report_trading_separately: bool = True
+
+
+@dataclass
+class SignalInfluenceTracker:
+    """Tracks canary signal influence in paper trading.
+
+    Reports extraction correctness separately from trading outcomes.
+    Reviews material divergences and tracks promotion readiness.
+    """
+
+    config: SignalInfluenceConfig
+    promotion_status: PromotionStatus = PromotionStatus.PENDING
+    _divergences: list[DivergenceRecord] = field(default_factory=list)
+    _extraction_metrics: dict[str, float] = field(default_factory=dict)
+    _trading_metrics: dict[str, float] = field(default_factory=dict)
+    _total_signals: int = 0
+    _v3_signals: int = 0
+    _approval_timestamp: datetime | None = None
+    _approver_id: str = ""
+
+    def start_paper_trading(self) -> None:
+        """Begin paper trading with v3 signals."""
+        self.config.enabled = True
+        self.promotion_status = PromotionStatus.PAPER_TRADING
+
+    def record_signal(self, is_v3: bool = False) -> None:
+        """Record a signal processed."""
+        self._total_signals += 1
+        if is_v3:
+            self._v3_signals += 1
+
+    def record_divergence(self, divergence: DivergenceRecord) -> None:
+        """Record a material recommendation divergence."""
+        self._divergences.append(divergence)
+
+    def update_extraction_metrics(self, metrics: dict[str, float]) -> None:
+        """Update extraction correctness metrics (separate from trading)."""
+        self._extraction_metrics.update(metrics)
+
+    def update_trading_metrics(self, metrics: dict[str, float]) -> None:
+        """Update trading outcome metrics (separate from extraction)."""
+        self._trading_metrics.update(metrics)
+
+    @property
+    def divergence_rate(self) -> float:
+        if self._v3_signals == 0:
+            return 0.0
+        return len(self._divergences) / self._v3_signals
+
+    @property
+    def unreviewed_divergences(self) -> list[DivergenceRecord]:
+        return [d for d in self._divergences if not d.reviewed]
+
+    def request_approval(self) -> None:
+        """Move to awaiting review status."""
+        self.promotion_status = PromotionStatus.AWAITING_REVIEW
+
+    def approve(self, approver_id: str) -> bool:
+        """Approve promotion. Requires owner approval if configured.
+
+        Returns False if approval requirements are not met.
+        """
+        if self.config.require_owner_approval:
+            if not approver_id:
+                return False
+            if self.config.owner_id and approver_id != self.config.owner_id:
+                return False
+
+        # Check all gates
+        if not self._all_gates_pass():
+            return False
+
+        self.promotion_status = PromotionStatus.APPROVED
+        self._approval_timestamp = datetime.now(timezone.utc)
+        self._approver_id = approver_id
+        return True
+
+    def reject(self, reason: str = "") -> None:
+        """Reject promotion."""
+        self.promotion_status = PromotionStatus.REJECTED
+
+    def _all_gates_pass(self) -> bool:
+        """Check if extraction correctness gates pass.
+
+        Trading outcomes explicitly do NOT override correctness gates
+        (Requirement 16.10).
+        """
+        # Divergence rate must be below threshold
+        if self.divergence_rate > self.config.max_divergence_rate:
+            return False
+
+        # All divergences must be reviewed
+        if self.unreviewed_divergences:
+            return False
+
+        return True
+
+    def summary(self) -> dict[str, Any]:
+        return {
+            "enabled": self.config.enabled,
+            "status": self.promotion_status.value,
+            "percentage": self.config.percentage,
+            "total_signals": self._total_signals,
+            "v3_signals": self._v3_signals,
+            "divergence_count": len(self._divergences),
+            "divergence_rate": self.divergence_rate,
+            "unreviewed_divergences": len(self.unreviewed_divergences),
+            "extraction_metrics": self._extraction_metrics,
+            "trading_metrics": self._trading_metrics,
+        }

services/intelligence_pipeline_v3/canary/routing.py

@@ -0,0 +1,260 @@
+"""Canary compatibility outputs — percentage routing and automatic rollback.
+
+Enables v3 adapter outputs for non-trading consumers first, then
+progressively routes more traffic. Automatic rollback triggers on
+correctness, latency, queue, or availability thresholds. Rollback
+preserves v3 audit records.
+"""
+
+from __future__ import annotations
+
+import enum
+import hashlib
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID, uuid4
+
+
+class RollbackReason(str, enum.Enum):
+    """Reasons for automatic canary rollback."""
+
+    CORRECTNESS_THRESHOLD = "correctness_threshold"
+    LATENCY_THRESHOLD = "latency_threshold"
+    QUEUE_SATURATION = "queue_saturation"
+    AVAILABILITY_THRESHOLD = "availability_threshold"
+    ERROR_RATE = "error_rate"
+    MANUAL = "manual"
+
+
+@dataclass(frozen=True)
+class RollbackEvent:
+    """Immutable record of a canary rollback.
+
+    Rollback leaves v3 audit records intact — only routing changes.
+    """
+
+    event_id: UUID
+    timestamp: datetime
+    reason: RollbackReason
+    previous_percentage: int
+    metric_value: float
+    threshold_value: float
+    details: str = ""
+
+    @classmethod
+    def create(
+        cls,
+        reason: RollbackReason,
+        previous_percentage: int,
+        metric_value: float,
+        threshold_value: float,
+        details: str = "",
+    ) -> RollbackEvent:
+        return cls(
+            event_id=uuid4(),
+            timestamp=datetime.now(timezone.utc),
+            reason=reason,
+            previous_percentage=previous_percentage,
+            metric_value=metric_value,
+            threshold_value=threshold_value,
+            details=details,
+        )
+
+
+@dataclass
+class CanaryConfig:
+    """Canary routing configuration with thresholds."""
+
+    enabled: bool = False
+    percentage: int = 0  # 0-100, percentage of docs using v3 outputs
+    document_types: set[str] = field(default_factory=set)  # Types eligible for canary
+    exclude_trading: bool = True  # Exclude trading consumers initially
+
+    # Automatic rollback thresholds
+    max_error_rate: float = 0.05
+    max_p95_latency_ms: float = 5000.0
+    max_queue_saturation: float = 0.90
+    min_availability: float = 0.95
+    min_correctness: float = 0.90
+
+    # Rollback behavior
+    rollback_to_percentage: int = 0  # Roll back to this percentage
+    cooldown_minutes: int = 60  # Wait before re-enabling after rollback
+
+
+@dataclass
+class CanaryRouter:
+    """Routes documents between v2 and v3 outputs at configurable percentages.
+
+    Routing is deterministic per document_id to avoid inconsistent
+    behavior on retries. Rollback preserves all v3 audit records.
+    """
+
+    config: CanaryConfig
+    _rollback_events: list[RollbackEvent] = field(default_factory=list)
+    _documents_routed_v3: int = 0
+    _documents_routed_v2: int = 0
+    _last_rollback: datetime | None = None
+
+    def should_use_v3(
+        self,
+        document_id: str,
+        document_type: str | None = None,
+        is_trading_consumer: bool = False,
+    ) -> bool:
+        """Determine if a document should use v3 outputs.
+
+        Deterministic per document_id for consistency.
+        """
+        if not self.config.enabled:
+            return False
+
+        # Respect trading exclusion
+        if is_trading_consumer and self.config.exclude_trading:
+            return False
+
+        # Check if in cooldown after rollback
+        if self._in_cooldown():
+            return False
+
+        # Document type filter
+        if (
+            self.config.document_types
+            and document_type
+            and document_type not in self.config.document_types
+        ):
+            return False
+
+        # Percentage-based routing (deterministic hash)
+        bucket = self._hash_to_bucket(document_id)
+        use_v3 = bucket < self.config.percentage
+
+        if use_v3:
+            self._documents_routed_v3 += 1
+        else:
+            self._documents_routed_v2 += 1
+
+        return use_v3
+
+    def check_rollback(
+        self,
+        error_rate: float = 0.0,
+        p95_latency_ms: float = 0.0,
+        queue_saturation: float = 0.0,
+        availability: float = 1.0,
+        correctness: float = 1.0,
+    ) -> RollbackEvent | None:
+        """Check all rollback thresholds. Returns event if rollback triggered."""
+        if not self.config.enabled or self.config.percentage == 0:
+            return None
+
+        checks: list[tuple[RollbackReason, float, float, str]] = [
+            (
+                RollbackReason.ERROR_RATE,
+                error_rate,
+                self.config.max_error_rate,
+                f"Error rate {error_rate:.3f} > {self.config.max_error_rate}",
+            ),
+            (
+                RollbackReason.LATENCY_THRESHOLD,
+                p95_latency_ms,
+                self.config.max_p95_latency_ms,
+                f"P95 latency {p95_latency_ms:.0f}ms > {self.config.max_p95_latency_ms:.0f}ms",
+            ),
+            (
+                RollbackReason.QUEUE_SATURATION,
+                queue_saturation,
+                self.config.max_queue_saturation,
+                f"Queue saturation {queue_saturation:.2f} > {self.config.max_queue_saturation}",
+            ),
+        ]
+
+        for reason, value, threshold, details in checks:
+            if value > threshold:
+                return self._trigger_rollback(reason, value, threshold, details)
+
+        # These check for below threshold
+        if availability < self.config.min_availability:
+            return self._trigger_rollback(
+                RollbackReason.AVAILABILITY_THRESHOLD,
+                availability,
+                self.config.min_availability,
+                f"Availability {availability:.3f} < {self.config.min_availability}",
+            )
+
+        if correctness < self.config.min_correctness:
+            return self._trigger_rollback(
+                RollbackReason.CORRECTNESS_THRESHOLD,
+                correctness,
+                self.config.min_correctness,
+                f"Correctness {correctness:.3f} < {self.config.min_correctness}",
+            )
+
+        return None
+
+    def manual_rollback(self, details: str = "") -> RollbackEvent:
+        """Trigger a manual rollback."""
+        return self._trigger_rollback(
+            RollbackReason.MANUAL,
+            0.0,
+            0.0,
+            details or "Manual rollback requested",
+        )
+
+    def _trigger_rollback(
+        self,
+        reason: RollbackReason,
+        metric_value: float,
+        threshold_value: float,
+        details: str,
+    ) -> RollbackEvent:
+        """Execute rollback — change routing but preserve audit data."""
+        event = RollbackEvent.create(
+            reason=reason,
+            previous_percentage=self.config.percentage,
+            metric_value=metric_value,
+            threshold_value=threshold_value,
+            details=details,
+        )
+        self.config.percentage = self.config.rollback_to_percentage
+        self._rollback_events.append(event)
+        self._last_rollback = datetime.now(timezone.utc)
+        return event
+
+    def _in_cooldown(self) -> bool:
+        """Check if we're in cooldown after a rollback."""
+        if self._last_rollback is None:
+            return False
+        from datetime import timedelta
+
+        cooldown_end = self._last_rollback + timedelta(
+            minutes=self.config.cooldown_minutes
+        )
+        return datetime.now(timezone.utc) < cooldown_end
+
+    def _hash_to_bucket(self, document_id: str) -> int:
+        """Deterministic hash to 0-99 bucket."""
+        h = hashlib.sha256(f"canary:{document_id}".encode()).hexdigest()
+        return int(h[:8], 16) % 100
+
+    @property
+    def rollback_events(self) -> list[RollbackEvent]:
+        return list(self._rollback_events)
+
+    @property
+    def v3_traffic_ratio(self) -> float:
+        total = self._documents_routed_v2 + self._documents_routed_v3
+        if total == 0:
+            return 0.0
+        return self._documents_routed_v3 / total
+
+    def summary(self) -> dict[str, Any]:
+        return {
+            "enabled": self.config.enabled,
+            "percentage": self.config.percentage,
+            "documents_v3": self._documents_routed_v3,
+            "documents_v2": self._documents_routed_v2,
+            "rollback_count": len(self._rollback_events),
+            "in_cooldown": self._in_cooldown(),
+        }

services/intelligence_pipeline_v3/compatibility/__init__.py

@@ -0,0 +1,20 @@
+"""Compatibility adapter — maps v3 intelligence records to current v2 data classes."""
+
+from services.intelligence_pipeline_v3.compatibility.adapter import CompatibilityAdapter
+from services.intelligence_pipeline_v3.compatibility.config import AdapterMode, is_adapter_enabled
+from services.intelligence_pipeline_v3.compatibility.models import (
+    AdapterLineage,
+    V2ImpactRecord,
+    V2IntelligenceRecord,
+    V3IntelligenceRecord,
+)
+
+__all__ = [
+    "AdapterLineage",
+    "AdapterMode",
+    "CompatibilityAdapter",
+    "V2ImpactRecord",
+    "V2IntelligenceRecord",
+    "V3IntelligenceRecord",
+    "is_adapter_enabled",
+]

services/intelligence_pipeline_v3/compatibility/adapter.py

@@ -0,0 +1,208 @@
+"""Compatibility adapter — maps approved v3 records to current v2 data classes.
+
+The adapter creates current-format records without discarding v3 provenance.
+It marks model_provider='hybrid' and stores complete stage lineage separately.
+
+Design reference: Section K (Compatibility Adapter) in design.md.
+"""
+
+from __future__ import annotations
+
+import uuid
+
+from services.intelligence_pipeline_v3.compatibility.config import (
+    AdapterMode,
+    is_adapter_enabled,
+)
+from services.intelligence_pipeline_v3.compatibility.models import (
+    AdapterLineage,
+    V2ImpactRecord,
+    V2IntelligenceRecord,
+    V3CompanySignal,
+    V3HorizonProbabilities,
+    V3IntelligenceRecord,
+    V3SentimentDistribution,
+)
+
+ADAPTER_VERSION = "1.0.0"
+
+
+class AdapterDisabledError(Exception):
+    """Raised when the adapter is called in disabled mode."""
+
+    pass
+
+
+class CompatibilityAdapter:
+    """Maps v3 intelligence records to v2 format for downstream consumers.
+
+    The adapter is gated by AdapterMode — it refuses to produce output when
+    disabled, ensuring v3 records cannot accidentally affect production
+    consumers until explicitly enabled.
+    """
+
+    def __init__(self, mode: AdapterMode = AdapterMode.DISABLED) -> None:
+        self._mode = mode
+
+    @property
+    def mode(self) -> AdapterMode:
+        return self._mode
+
+    @property
+    def version(self) -> str:
+        return ADAPTER_VERSION
+
+    def map_to_v2(
+        self, v3_record: V3IntelligenceRecord
+    ) -> tuple[V2IntelligenceRecord, AdapterLineage]:
+        """Map an approved v3 record to v2 intelligence + impact records.
+
+        Returns:
+            A tuple of (V2IntelligenceRecord, AdapterLineage).
+
+        Raises:
+            AdapterDisabledError: If the adapter is in disabled mode.
+        """
+        if not is_adapter_enabled(self._mode):
+            raise AdapterDisabledError(
+                f"Adapter is disabled (mode={self._mode.value}). "
+                "Enable replay, shadow, canary, or production mode to use."
+            )
+
+        v2_id = str(uuid.uuid4())
+
+        # Map each company signal to a v2 impact record
+        impact_records = [
+            self._map_company_signal(signal) for signal in v3_record.company_signals
+        ]
+
+        v2_record = V2IntelligenceRecord(
+            id=v2_id,
+            document_id=v3_record.document_id,
+            summary=v3_record.summary,
+            macro_themes=v3_record.macro_themes,
+            novelty_score=v3_record.novelty_score,
+            confidence=v3_record.confidence,
+            model_provider="hybrid",
+            model_name="intelligence-pipeline-v3",
+            prompt_version=f"adapter-{ADAPTER_VERSION}",
+            schema_version="3.0.0",
+            impact_records=impact_records,
+        )
+
+        lineage = AdapterLineage(
+            adapter_version=ADAPTER_VERSION,
+            pipeline_version=v3_record.pipeline_version,
+            v3_document_id=v3_record.document_id,
+            v2_intelligence_id=v2_id,
+            stage_runs=v3_record.stage_runs,
+            mapping_notes=[
+                f"Mapped {len(v3_record.company_signals)} company signals",
+                f"Mode: {self._mode.value}",
+            ],
+        )
+
+        return v2_record, lineage
+
+    def _map_company_signal(self, signal: V3CompanySignal) -> V2ImpactRecord:
+        """Map a single v3 company signal to a v2 impact record."""
+        return V2ImpactRecord(
+            company_id=signal.company_id,
+            ticker=signal.ticker,
+            relevance=signal.relevance_probability,
+            sentiment=self._map_sentiment(signal.sentiment),
+            impact_score=self._map_impact_score(signal),
+            impact_horizon=self._map_horizon(signal.horizon_probabilities),
+            catalyst_type=self._map_catalyst_type(signal.event_classes),
+            evidence_spans=signal.evidence_spans,
+        )
+
+    @staticmethod
+    def _map_sentiment(dist: V3SentimentDistribution) -> str:
+        """Map probability distribution to legacy sentiment enum.
+
+        Logic:
+        - If max probability is neutral and ≥ 0.5 → neutral
+        - If positive and negative are both ≥ 0.3 → mixed
+        - Otherwise take the argmax of positive/negative/neutral
+        """
+        pos, neg, neu = dist.positive, dist.negative, dist.neutral
+
+        # Mixed detection: both positive and negative have significant mass
+        if pos >= 0.3 and neg >= 0.3:
+            return "mixed"
+
+        # Argmax
+        max_val = max(pos, neg, neu)
+        if max_val == neu:
+            return "neutral"
+        elif max_val == pos:
+            return "positive"
+        else:
+            return "negative"
+
+    @staticmethod
+    def _map_impact_score(signal: V3CompanySignal) -> float:
+        """Map v3 expected_magnitude to legacy impact_score in [-1, 1].
+
+        The v3 expected_magnitude is already a signed value representing
+        expected market response. We clamp to [-1, 1] for legacy compatibility.
+
+        If expected_magnitude is None, derive a conservative estimate from
+        direction probabilities.
+        """
+        if signal.expected_magnitude is not None:
+            return max(-1.0, min(1.0, signal.expected_magnitude))
+
+        # Fallback: derive from direction probabilities
+        dp = signal.direction_probabilities
+        # Signed score: positive_prob - negative_prob, scaled to [-1, 1]
+        signed = dp.positive - dp.negative
+        return max(-1.0, min(1.0, signed))
+
+    @staticmethod
+    def _map_horizon(probs: V3HorizonProbabilities) -> str:
+        """Map horizon probability distribution to single legacy horizon string.
+
+        Returns the horizon with the highest probability (argmax).
+        Ties are broken by preferring shorter horizons.
+        """
+        horizon_map = {
+            "intraday": probs.intraday,
+            "1d": probs.one_day,
+            "7d": probs.seven_day,
+            "30d": probs.thirty_day,
+            "90d": probs.ninety_day,
+        }
+
+        # argmax with tie-breaking by order (shortest first)
+        return max(horizon_map, key=lambda k: horizon_map[k])
+
+    @staticmethod
+    def _map_catalyst_type(event_classes: list[str]) -> str:
+        """Map v3 event taxonomy to legacy catalyst_type enum.
+
+        Uses the first matching event class. Falls back to 'other'.
+        """
+        # Mapping from v3 event classes to legacy CatalystType values
+        event_to_catalyst: dict[str, str] = {
+            "earnings_beat": "earnings",
+            "earnings_miss": "earnings",
+            "guidance_raise": "earnings",
+            "guidance_cut": "earnings",
+            "product_launch": "product",
+            "legal_regulatory": "legal",
+            "ma_announcement": "m_and_a",
+            "supply_chain": "supply_chain",
+            "rating_change": "rating_change",
+            "macro_event": "macro",
+            "management_change": "other",
+            "dividend_change": "other",
+            "buyback": "other",
+        }
+
+        for event_class in event_classes:
+            if event_class in event_to_catalyst:
+                return event_to_catalyst[event_class]
+
+        return "other"

services/intelligence_pipeline_v3/compatibility/config.py

@@ -0,0 +1,39 @@
+"""Feature flag configuration for the compatibility adapter.
+
+The adapter is disabled by default and must be explicitly enabled for
+replay, shadow, canary, or production modes.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+
+
+class AdapterMode(str, Enum):
+    """Operating mode for the compatibility adapter.
+
+    - disabled: adapter does not run (default)
+    - replay_only: adapter runs during offline replay evaluation
+    - shadow_only: adapter runs in shadow mode (no downstream effect)
+    - canary: adapter outputs routed to a percentage of non-trading consumers
+    - production: adapter outputs used for all consumers
+    """
+
+    DISABLED = "disabled"
+    REPLAY_ONLY = "replay_only"
+    SHADOW_ONLY = "shadow_only"
+    CANARY = "canary"
+    PRODUCTION = "production"
+
+
+def is_adapter_enabled(mode: AdapterMode) -> bool:
+    """Return True if the adapter should produce output in the given mode.
+
+    Only replay, shadow, canary, and production modes enable output.
+    The disabled mode prevents any adapter execution.
+    """
+    return mode != AdapterMode.DISABLED
+
+
+# Default mode — adapter is OFF until explicitly activated
+DEFAULT_ADAPTER_MODE: AdapterMode = AdapterMode.DISABLED

services/intelligence_pipeline_v3/compatibility/models.py

@@ -0,0 +1,162 @@
+"""Input/output models for the v3→v2 compatibility adapter.
+
+V3IntelligenceRecord represents the full v3 pipeline output.
+V2IntelligenceRecord / V2ImpactRecord match the current document_intelligence
+and document_impact_records database schemas.
+AdapterLineage captures version and stage provenance.
+"""
+
+from __future__ import annotations
+
+import uuid
+from datetime import datetime, timezone
+from typing import Literal
+
+from pydantic import BaseModel, Field
+
+# ---------------------------------------------------------------------------
+# V3 Pipeline Output (input to adapter)
+# ---------------------------------------------------------------------------
+
+
+class V3SentimentDistribution(BaseModel):
+    """Per-company calibrated sentiment probabilities."""
+
+    positive: float = Field(ge=0.0, le=1.0)
+    negative: float = Field(ge=0.0, le=1.0)
+    neutral: float = Field(ge=0.0, le=1.0)
+
+
+class V3HorizonProbabilities(BaseModel):
+    """Probability distribution over impact horizons."""
+
+    intraday: float = Field(ge=0.0, le=1.0, default=0.0)
+    one_day: float = Field(ge=0.0, le=1.0, default=0.0)
+    seven_day: float = Field(ge=0.0, le=1.0, default=0.0)
+    thirty_day: float = Field(ge=0.0, le=1.0, default=0.0)
+    ninety_day: float = Field(ge=0.0, le=1.0, default=0.0)
+
+
+class V3DirectionProbabilities(BaseModel):
+    """Probability distribution over market direction."""
+
+    positive: float = Field(ge=0.0, le=1.0, default=0.0)
+    negative: float = Field(ge=0.0, le=1.0, default=0.0)
+    neutral: float = Field(ge=0.0, le=1.0, default=0.0)
+
+
+class V3CompanySignal(BaseModel):
+    """A single company's signal from the v3 pipeline."""
+
+    company_id: str
+    ticker: str
+    relevance_probability: float = Field(ge=0.0, le=1.0)
+    event_classes: list[str] = Field(default_factory=list)
+    sentiment: V3SentimentDistribution
+    direction_probabilities: V3DirectionProbabilities
+    horizon_probabilities: V3HorizonProbabilities
+    expected_magnitude: float | None = None
+    evidence_spans: list[str] = Field(default_factory=list)
+    adjudicated: bool = False
+
+
+class V3StageRun(BaseModel):
+    """Lineage for a single pipeline stage execution."""
+
+    stage: str
+    endpoint_id: str | None = None
+    deployment_id: str | None = None
+    model_version: str | None = None
+    schema_version: str = "1.0.0"
+    calibration_version: str | None = None
+    started_at: datetime = Field(default_factory=lambda: datetime.now(tz=timezone.utc))
+    duration_ms: int = 0
+    status: str = "completed"
+
+
+class V3IntelligenceRecord(BaseModel):
+    """Complete v3 pipeline output for a single document.
+
+    This is the adapter's input — the full v3 record with probabilities,
+    evidence, and stage lineage.
+    """
+
+    document_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
+    document_type: str = "article"
+    summary: str = ""
+    macro_themes: list[str] = Field(default_factory=list)
+    novelty_score: float = Field(ge=0.0, le=1.0, default=0.5)
+    confidence: float = Field(ge=0.0, le=1.0, default=0.5)
+    company_signals: list[V3CompanySignal] = Field(default_factory=list)
+    stage_runs: list[V3StageRun] = Field(default_factory=list)
+    pipeline_version: str = "3.0.0"
+    created_at: datetime = Field(default_factory=lambda: datetime.now(tz=timezone.utc))
+
+
+# ---------------------------------------------------------------------------
+# V2 Output (adapter output — matches current DB schema)
+# ---------------------------------------------------------------------------
+
+
+class V2ImpactRecord(BaseModel):
+    """Maps to document_impact_records table.
+
+    Fields match the columns: relevance, sentiment (enum string),
+    impact_score (float), impact_horizon (string), catalyst_type,
+    key_facts, risks, evidence_spans.
+    """
+
+    id: str = Field(default_factory=lambda: str(uuid.uuid4()))
+    company_id: str
+    ticker: str
+    relevance: float = Field(ge=0.0, le=1.0)
+    sentiment: Literal["positive", "negative", "neutral", "mixed"]
+    impact_score: float = Field(ge=-1.0, le=1.0)
+    impact_horizon: Literal["intraday", "1d", "7d", "30d", "90d"]
+    catalyst_type: str = "other"
+    key_facts: list[str] = Field(default_factory=list)
+    risks: list[str] = Field(default_factory=list)
+    evidence_spans: list[str] = Field(default_factory=list)
+
+
+class V2IntelligenceRecord(BaseModel):
+    """Maps to document_intelligence table.
+
+    Fields match columns: summary, macro_themes, novelty_score,
+    source_credibility, confidence, model_provider, model_name,
+    prompt_version, schema_version, plus associated impact records.
+    """
+
+    id: str = Field(default_factory=lambda: str(uuid.uuid4()))
+    document_id: str
+    summary: str = ""
+    macro_themes: list[str] = Field(default_factory=list)
+    novelty_score: float = Field(ge=0.0, le=1.0)
+    source_credibility: float = Field(ge=0.0, le=1.0, default=0.5)
+    confidence: float = Field(ge=0.0, le=1.0)
+    model_provider: str = "hybrid"
+    model_name: str = "intelligence-pipeline-v3"
+    prompt_version: str = ""
+    schema_version: str = "3.0.0"
+    impact_records: list[V2ImpactRecord] = Field(default_factory=list)
+    created_at: datetime = Field(default_factory=lambda: datetime.now(tz=timezone.utc))
+
+
+# ---------------------------------------------------------------------------
+# Adapter Lineage
+# ---------------------------------------------------------------------------
+
+
+class AdapterLineage(BaseModel):
+    """Records which adapter version produced the v2 record and from what v3 data.
+
+    Stored separately so v3 provenance is never lost.
+    """
+
+    adapter_version: str = "1.0.0"
+    pipeline_version: str = "3.0.0"
+    v3_document_id: str
+    v2_intelligence_id: str
+    stage_runs: list[V3StageRun] = Field(default_factory=list)
+    mapped_at: datetime = Field(default_factory=lambda: datetime.now(tz=timezone.utc))
+    mapping_notes: list[str] = Field(default_factory=list)

services/intelligence_pipeline_v3/confidence/__init__.py

@@ -0,0 +1,32 @@
+"""Confidence feature pipeline for Intelligence Pipeline v3.
+
+Provides calibrated extraction confidence from specialist scores,
+symbol resolution, evidence validation, schema completeness,
+model agreement, and historical calibration data. Replaces
+generative model self-reported confidence with empirically
+calibrated probabilities.
+"""
+
+from services.intelligence_pipeline_v3.confidence.artifacts import (
+    load_artifact,
+    save_artifact,
+)
+from services.intelligence_pipeline_v3.confidence.calibrator import ConfidenceCalibrator
+from services.intelligence_pipeline_v3.confidence.defaults import get_default_confidence
+from services.intelligence_pipeline_v3.confidence.features import ConfidenceFeatureExtractor
+from services.intelligence_pipeline_v3.confidence.models import (
+    CalibrationArtifactMetadata,
+    ConfidenceFeatures,
+    ConfidenceResult,
+)
+
+__all__ = [
+    "CalibrationArtifactMetadata",
+    "ConfidenceCalibrator",
+    "ConfidenceFeatureExtractor",
+    "ConfidenceFeatures",
+    "ConfidenceResult",
+    "get_default_confidence",
+    "load_artifact",
+    "save_artifact",
+]

services/intelligence_pipeline_v3/confidence/artifacts.py

@@ -0,0 +1,186 @@
+"""Calibration artifact persistence.
+
+Handles versioned save/load of fitted calibrator objects alongside
+metadata including training provenance, quality metrics, and version.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import pickle
+from pathlib import Path
+
+from services.intelligence_pipeline_v3.confidence.calibrator import ConfidenceCalibrator
+from services.intelligence_pipeline_v3.confidence.models import CalibrationArtifactMetadata
+
+logger = logging.getLogger(__name__)
+
+ARTIFACT_FILE = "calibrator.pkl"
+METADATA_FILE = "metadata.json"
+
+
+def save_artifact(
+    calibrator: ConfidenceCalibrator,
+    version: str,
+    path: str | Path,
+) -> Path:
+    """Save a fitted calibrator and metadata to a versioned directory.
+
+    Creates the directory structure:
+        <path>/<version>/calibrator.pkl
+        <path>/<version>/metadata.json
+
+    Parameters
+    ----------
+    calibrator
+        A fitted ConfidenceCalibrator instance.
+    version
+        Version string for this artifact (e.g., "v1.0.0").
+    path
+        Base directory for artifact storage.
+
+    Returns
+    -------
+    Path
+        Path to the versioned artifact directory.
+
+    Raises
+    ------
+    ValueError
+        If the calibrator has not been fitted.
+    """
+    if not calibrator.is_fitted:
+        raise ValueError("Cannot save an unfitted calibrator")
+
+    artifact_dir = Path(path) / version
+    artifact_dir.mkdir(parents=True, exist_ok=True)
+
+    # Save the calibrator model
+    calibrator_path = artifact_dir / ARTIFACT_FILE
+    with open(calibrator_path, "wb") as f:
+        pickle.dump(calibrator, f, protocol=pickle.HIGHEST_PROTOCOL)
+
+    # Save metadata
+    metadata = calibrator.metadata
+    if metadata is None:
+        metadata = CalibrationArtifactMetadata(
+            version=version,
+            method=calibrator.method,  # type: ignore[arg-type]
+            training_count=0,
+            training_range="unknown",
+            ece=0.0,
+            brier_score=0.0,
+        )
+
+    metadata_path = artifact_dir / METADATA_FILE
+    with open(metadata_path, "w") as f:
+        json.dump(metadata.model_dump(mode="json"), f, indent=2, default=str)
+
+    logger.info(
+        "Saved calibration artifact: version=%s, method=%s, path=%s",
+        version,
+        calibrator.method,
+        artifact_dir,
+    )
+    return artifact_dir
+
+
+def load_artifact(path: str | Path) -> ConfidenceCalibrator:
+    """Load a calibrator from a versioned artifact directory.
+
+    Expects the directory to contain calibrator.pkl and metadata.json.
+
+    Parameters
+    ----------
+    path
+        Path to the versioned artifact directory (e.g., <base>/v1.0.0/).
+
+    Returns
+    -------
+    ConfidenceCalibrator
+        The loaded and ready-to-use calibrator.
+
+    Raises
+    ------
+    FileNotFoundError
+        If the artifact directory or files don't exist.
+    ValueError
+        If the loaded object is not a ConfidenceCalibrator.
+    """
+    artifact_dir = Path(path)
+
+    calibrator_path = artifact_dir / ARTIFACT_FILE
+    if not calibrator_path.exists():
+        raise FileNotFoundError(
+            f"Calibrator artifact not found at {calibrator_path}"
+        )
+
+    with open(calibrator_path, "rb") as f:
+        calibrator = pickle.load(f)  # noqa: S301
+
+    if not isinstance(calibrator, ConfidenceCalibrator):
+        raise ValueError(
+            f"Loaded object is not a ConfidenceCalibrator: {type(calibrator)}"
+        )
+
+    logger.info(
+        "Loaded calibration artifact: version=%s, method=%s, path=%s",
+        calibrator.version,
+        calibrator.method,
+        artifact_dir,
+    )
+    return calibrator
+
+
+def load_metadata(path: str | Path) -> CalibrationArtifactMetadata:
+    """Load only the metadata for an artifact without loading the full model.
+
+    Parameters
+    ----------
+    path
+        Path to the versioned artifact directory.
+
+    Returns
+    -------
+    CalibrationArtifactMetadata
+        The artifact metadata.
+
+    Raises
+    ------
+    FileNotFoundError
+        If the metadata file doesn't exist.
+    """
+    metadata_path = Path(path) / METADATA_FILE
+    if not metadata_path.exists():
+        raise FileNotFoundError(f"Metadata not found at {metadata_path}")
+
+    with open(metadata_path) as f:
+        data = json.load(f)
+
+    return CalibrationArtifactMetadata(**data)
+
+
+def list_versions(base_path: str | Path) -> list[str]:
+    """List all available artifact versions in a base directory.
+
+    Parameters
+    ----------
+    base_path
+        Base directory containing versioned subdirectories.
+
+    Returns
+    -------
+    list[str]
+        Sorted list of version strings.
+    """
+    base = Path(base_path)
+    if not base.exists():
+        return []
+
+    versions = []
+    for item in base.iterdir():
+        if item.is_dir() and (item / ARTIFACT_FILE).exists():
+            versions.append(item.name)
+
+    return sorted(versions)

services/intelligence_pipeline_v3/confidence/calibrator.py

@@ -0,0 +1,406 @@
+"""Confidence calibrator using isotonic or Platt scaling.
+
+Maps confidence feature vectors to calibrated correctness probabilities.
+Supports training on held-out Gold_Corpus data, cross-validation for
+method comparison, and versioned artifact tracking.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Literal
+
+import numpy as np
+
+from services.intelligence_pipeline_v3.confidence.models import (
+    CalibrationArtifactMetadata,
+    ConfidenceFeatures,
+)
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_VERSION = "uncalibrated"
+
+
+class ConfidenceCalibrator:
+    """Calibrates confidence features to correctness probabilities.
+
+    Supports isotonic regression and Platt (logistic) scaling.
+    The calibrator is fitted on labeled Gold_Corpus data where labels
+    indicate whether the extraction was correct (True) or not (False).
+
+    Parameters
+    ----------
+    method
+        Calibration method: "isotonic" for non-parametric monotone fit,
+        "platt" for logistic regression scaling.
+    """
+
+    def __init__(self, method: Literal["isotonic", "platt"] = "isotonic") -> None:
+        self._method: Literal["isotonic", "platt"] = method
+        self._version: str = DEFAULT_VERSION
+        self._fitted: bool = False
+        self._model: object | None = None
+        self._metadata: CalibrationArtifactMetadata | None = None
+        self._training_count: int = 0
+
+    @property
+    def method(self) -> str:
+        """Return the calibration method."""
+        return self._method
+
+    @property
+    def version(self) -> str:
+        """Return the calibration artifact version."""
+        return self._version
+
+    @property
+    def is_fitted(self) -> bool:
+        """Return whether the calibrator has been fitted."""
+        return self._fitted
+
+    @property
+    def metadata(self) -> CalibrationArtifactMetadata | None:
+        """Return the artifact metadata if fitted."""
+        return self._metadata
+
+    def fit(
+        self,
+        features: list[ConfidenceFeatures],
+        labels: list[bool],
+        method: str | None = None,
+        version: str = "v1.0.0",
+        training_range: str = "unknown",
+    ) -> None:
+        """Train the calibrator on labeled feature/correctness pairs.
+
+        Parameters
+        ----------
+        features
+            List of confidence feature vectors from training data.
+        labels
+            True if the extraction was correct, False otherwise.
+        method
+            Override method for this fit (isotonic or platt).
+            If None, uses the instance default.
+        version
+            Version string for the resulting artifact.
+        training_range
+            Description of the training data date range.
+
+        Raises
+        ------
+        ValueError
+            If features and labels have different lengths or are empty.
+        """
+        if not features or not labels:
+            raise ValueError("features and labels must not be empty")
+        if len(features) != len(labels):
+            raise ValueError(
+                f"features ({len(features)}) and labels ({len(labels)}) must have the same length"
+            )
+
+        if method is not None:
+            if method not in ("isotonic", "platt"):
+                raise ValueError(f"method must be 'isotonic' or 'platt', got '{method}'")
+            self._method = method  # type: ignore[assignment]
+
+        # Convert features to matrix
+        X = np.array([f.to_vector() for f in features], dtype=np.float64)
+        y = np.array(labels, dtype=np.float64)
+
+        if self._method == "isotonic":
+            self._fit_isotonic(X, y)
+        else:
+            self._fit_platt(X, y)
+
+        self._version = version
+        self._training_count = len(features)
+        self._fitted = True
+
+        # Compute calibration quality on training data (for metadata)
+        predictions = self._predict_batch(X)
+        ece = _compute_ece(predictions, y)
+        brier = _compute_brier(predictions, y)
+
+        self._metadata = CalibrationArtifactMetadata(
+            version=version,
+            method=self._method,
+            training_count=len(features),
+            training_range=training_range,
+            ece=ece,
+            brier_score=brier,
+        )
+
+        logger.info(
+            "ConfidenceCalibrator fitted: method=%s, n=%d, version=%s, ECE=%.4f, Brier=%.4f",
+            self._method,
+            len(features),
+            version,
+            ece,
+            brier,
+        )
+
+    def predict(self, features: ConfidenceFeatures) -> float:
+        """Return calibrated probability of extraction correctness.
+
+        Parameters
+        ----------
+        features
+            Confidence feature vector for a single extraction.
+
+        Returns
+        -------
+        float
+            Calibrated probability in [0, 1].
+        """
+        if not self._fitted:
+            # Return a neutral default when uncalibrated
+            return 0.5
+
+        X = np.array([features.to_vector()], dtype=np.float64)
+        predictions = self._predict_batch(X)
+        return float(np.clip(predictions[0], 0.0, 1.0))
+
+    def predict_batch(self, features_list: list[ConfidenceFeatures]) -> list[float]:
+        """Return calibrated probabilities for a batch of feature vectors.
+
+        Parameters
+        ----------
+        features_list
+            List of confidence feature vectors.
+
+        Returns
+        -------
+        list[float]
+            Calibrated probabilities in [0, 1].
+        """
+        if not self._fitted:
+            return [0.5] * len(features_list)
+
+        X = np.array([f.to_vector() for f in features_list], dtype=np.float64)
+        predictions = self._predict_batch(X)
+        return [float(np.clip(p, 0.0, 1.0)) for p in predictions]
+
+    def evaluate(
+        self,
+        features: list[ConfidenceFeatures],
+        labels: list[bool],
+    ) -> tuple[float, float]:
+        """Evaluate ECE and Brier score on held-out data.
+
+        Parameters
+        ----------
+        features
+            Held-out feature vectors.
+        labels
+            True correctness labels.
+
+        Returns
+        -------
+        tuple[float, float]
+            (ECE, Brier_score) on the held-out set.
+        """
+        if not features or not labels:
+            raise ValueError("features and labels must not be empty")
+        if len(features) != len(labels):
+            raise ValueError("features and labels must have the same length")
+
+        X = np.array([f.to_vector() for f in features], dtype=np.float64)
+        y = np.array(labels, dtype=np.float64)
+
+        if self._fitted:
+            predictions = self._predict_batch(X)
+        else:
+            predictions = np.full(len(y), 0.5)
+
+        ece = _compute_ece(predictions, y)
+        brier = _compute_brier(predictions, y)
+        return ece, brier
+
+    def _fit_isotonic(self, X: np.ndarray, y: np.ndarray) -> None:
+        """Fit isotonic regression on aggregated feature scores."""
+        from sklearn.isotonic import IsotonicRegression
+
+        # Aggregate features into a single score for isotonic monotone fit
+        aggregated = X.mean(axis=1)
+        iso = IsotonicRegression(y_min=0.0, y_max=1.0, out_of_bounds="clip")
+        iso.fit(aggregated, y)
+        self._model = iso
+
+    def _fit_platt(self, X: np.ndarray, y: np.ndarray) -> None:
+        """Fit logistic regression (Platt scaling) on the full feature vector."""
+        from sklearn.linear_model import LogisticRegression
+
+        y_int = y.astype(np.int32)
+        if len(np.unique(y_int)) < 2:
+            # Not enough class diversity — store a dummy model
+            self._model = _ConstantPredictor(float(y.mean()))
+            return
+
+        lr = LogisticRegression(solver="lbfgs", max_iter=1000, C=1.0)
+        lr.fit(X, y_int)
+        self._model = lr
+
+    def _predict_batch(self, X: np.ndarray) -> np.ndarray:
+        """Internal prediction dispatch."""
+        if self._model is None:
+            return np.full(X.shape[0], 0.5)
+
+        if self._method == "isotonic":
+            # Isotonic uses aggregated score
+            aggregated = X.mean(axis=1)
+            return self._model.predict(aggregated)  # type: ignore[union-attr]
+        else:
+            # Platt uses full feature vector
+            if isinstance(self._model, _ConstantPredictor):
+                return self._model.predict(X)
+            return self._model.predict_proba(X)[:, 1]  # type: ignore[union-attr]
+
+
+class _ConstantPredictor:
+    """Fallback predictor when training data has only one class."""
+
+    def __init__(self, value: float) -> None:
+        self._value = value
+
+    def predict(self, X: np.ndarray) -> np.ndarray:
+        return np.full(X.shape[0], self._value)
+
+
+def _compute_ece(
+    predictions: np.ndarray,
+    labels: np.ndarray,
+    n_bins: int = 10,
+) -> float:
+    """Compute Expected Calibration Error.
+
+    Partitions predictions into equal-width bins and computes the
+    weighted average of |avg_predicted - avg_actual| per bin.
+
+    Parameters
+    ----------
+    predictions
+        Predicted probabilities.
+    labels
+        True binary labels (0 or 1).
+    n_bins
+        Number of equal-width bins.
+
+    Returns
+    -------
+    float
+        ECE value in [0, 1].
+    """
+    if len(predictions) == 0:
+        return 0.0
+
+    bin_boundaries = np.linspace(0.0, 1.0, n_bins + 1)
+    ece = 0.0
+    n = len(predictions)
+
+    for i in range(n_bins):
+        lower = bin_boundaries[i]
+        upper = bin_boundaries[i + 1]
+
+        if i == n_bins - 1:
+            # Include right boundary in last bin
+            mask = (predictions >= lower) & (predictions <= upper)
+        else:
+            mask = (predictions >= lower) & (predictions < upper)
+
+        bin_count = mask.sum()
+        if bin_count == 0:
+            continue
+
+        avg_predicted = predictions[mask].mean()
+        avg_actual = labels[mask].mean()
+        ece += (bin_count / n) * abs(avg_predicted - avg_actual)
+
+    return float(ece)
+
+
+def _compute_brier(predictions: np.ndarray, labels: np.ndarray) -> float:
+    """Compute Brier score (mean squared error of probability predictions).
+
+    Parameters
+    ----------
+    predictions
+        Predicted probabilities.
+    labels
+        True binary labels (0 or 1).
+
+    Returns
+    -------
+    float
+        Brier score in [0, 1].
+    """
+    if len(predictions) == 0:
+        return 0.0
+    return float(np.mean((predictions - labels) ** 2))
+
+
+def compare_methods(
+    features: list[ConfidenceFeatures],
+    labels: list[bool],
+    n_folds: int = 5,
+) -> dict[str, dict[str, float]]:
+    """Compare isotonic and Platt methods using k-fold cross-validation.
+
+    Parameters
+    ----------
+    features
+        Full set of training features.
+    labels
+        Full set of correctness labels.
+    n_folds
+        Number of cross-validation folds.
+
+    Returns
+    -------
+    dict
+        Mapping of method name to {"ece": float, "brier": float} averages.
+    """
+    if len(features) < n_folds * 2:
+        raise ValueError(
+            f"Need at least {n_folds * 2} samples for {n_folds}-fold CV, got {len(features)}"
+        )
+
+    results: dict[str, list[tuple[float, float]]] = {
+        "isotonic": [],
+        "platt": [],
+    }
+
+    indices = np.arange(len(features))
+    fold_size = len(features) // n_folds
+
+    for fold in range(n_folds):
+        val_start = fold * fold_size
+        val_end = val_start + fold_size if fold < n_folds - 1 else len(features)
+
+        val_indices = indices[val_start:val_end]
+        train_indices = np.concatenate([indices[:val_start], indices[val_end:]])
+
+        train_features = [features[i] for i in train_indices]
+        train_labels = [labels[i] for i in train_indices]
+        val_features = [features[i] for i in val_indices]
+        val_labels = [labels[i] for i in val_indices]
+
+        for method_name in ("isotonic", "platt"):
+            cal = ConfidenceCalibrator(method=method_name)  # type: ignore[arg-type]
+            cal.fit(
+                train_features,
+                train_labels,
+                version=f"cv-fold-{fold}",
+                training_range="cross-validation",
+            )
+            ece, brier = cal.evaluate(val_features, val_labels)
+            results[method_name].append((ece, brier))
+
+    return {
+        method: {
+            "ece": float(np.mean([r[0] for r in scores])),
+            "brier": float(np.mean([r[1] for r in scores])),
+        }
+        for method, scores in results.items()
+    }

services/intelligence_pipeline_v3/confidence/defaults.py

@@ -0,0 +1,142 @@
+"""Conservative confidence defaults for underrepresented classes.
+
+When calibration data is insufficient for a specific document type or
+event class, returns conservative values (0.3-0.5) and marks the result
+as under-calibrated per Requirement 10.7.
+"""
+
+from __future__ import annotations
+
+import logging
+
+from services.intelligence_pipeline_v3.confidence.models import ConfidenceResult
+
+logger = logging.getLogger(__name__)
+
+# Conservative default probabilities by document type.
+# These are intentionally low (0.3-0.5) to avoid overconfidence
+# when insufficient calibration data exists.
+_DOCUMENT_TYPE_DEFAULTS: dict[str, float] = {
+    "news": 0.45,
+    "filing": 0.40,
+    "transcript": 0.40,
+    "press_release": 0.45,
+    "macro_event": 0.35,
+    "unknown": 0.30,
+}
+
+# Conservative default probabilities by event class.
+# More complex or rare event types get lower defaults.
+_EVENT_CLASS_DEFAULTS: dict[str, float] = {
+    "earnings_beat": 0.50,
+    "earnings_miss": 0.50,
+    "guidance_raise": 0.45,
+    "guidance_cut": 0.45,
+    "merger_acquisition": 0.40,
+    "product_launch": 0.45,
+    "regulatory_action": 0.40,
+    "management_change": 0.45,
+    "legal_proceeding": 0.40,
+    "supply_chain": 0.35,
+    "rating_change": 0.45,
+    "dividend_change": 0.45,
+    "buyback": 0.45,
+    "macro_policy": 0.35,
+    "geopolitical": 0.30,
+    "sector_rotation": 0.35,
+    "unknown": 0.30,
+}
+
+# Features used when returning conservative defaults
+_DEFAULT_FEATURES_USED = [
+    "document_type_prior",
+    "event_class_prior",
+]
+
+
+def get_default_confidence(
+    document_type: str,
+    event_class: str,
+) -> ConfidenceResult:
+    """Return a conservative confidence result for underrepresented classes.
+
+    Used when calibration data is insufficient for the given document type
+    and event class combination. Returns conservative probabilities (0.3-0.5)
+    and marks the result as under-calibrated.
+
+    Parameters
+    ----------
+    document_type
+        The document type (news, filing, transcript, etc.).
+    event_class
+        The classified event type (earnings_beat, merger_acquisition, etc.).
+
+    Returns
+    -------
+    ConfidenceResult
+        A conservative confidence result with under_calibrated=True.
+    """
+    doc_default = _DOCUMENT_TYPE_DEFAULTS.get(
+        document_type, _DOCUMENT_TYPE_DEFAULTS["unknown"]
+    )
+    event_default = _EVENT_CLASS_DEFAULTS.get(
+        event_class, _EVENT_CLASS_DEFAULTS["unknown"]
+    )
+
+    # Take the minimum of document and event defaults for extra conservatism
+    probability = min(doc_default, event_default)
+
+    logger.debug(
+        "Using conservative default confidence: doc_type=%s (%.2f), event=%s (%.2f) -> %.2f",
+        document_type,
+        doc_default,
+        event_class,
+        event_default,
+        probability,
+    )
+
+    return ConfidenceResult(
+        probability=probability,
+        features_used=_DEFAULT_FEATURES_USED,
+        is_calibrated=False,
+        under_calibrated=True,
+        calibration_version="conservative-default-v1",
+    )
+
+
+def is_underrepresented(
+    document_type: str,
+    event_class: str,
+    min_samples: int = 30,
+    known_counts: dict[tuple[str, str], int] | None = None,
+) -> bool:
+    """Check if a document_type + event_class combination is underrepresented.
+
+    Parameters
+    ----------
+    document_type
+        The document type.
+    event_class
+        The event class.
+    min_samples
+        Minimum number of calibration samples to consider a class well-represented.
+    known_counts
+        Optional mapping of (doc_type, event_class) -> sample count.
+        If None, treats any unknown combination as underrepresented.
+
+    Returns
+    -------
+    bool
+        True if the class has insufficient calibration data.
+    """
+    if known_counts is None:
+        # Without explicit counts, use heuristic: unknown types are underrepresented
+        if document_type not in _DOCUMENT_TYPE_DEFAULTS:
+            return True
+        if event_class not in _EVENT_CLASS_DEFAULTS:
+            return True
+        return False
+
+    key = (document_type, event_class)
+    count = known_counts.get(key, 0)
+    return count < min_samples

services/intelligence_pipeline_v3/confidence/features.py

@@ -0,0 +1,221 @@
+"""Confidence feature extraction from upstream pipeline stages.
+
+Computes field-level features from extraction, resolution, evidence,
+sentiment, and cross-stage agreement to produce a ConfidenceFeatures
+vector for calibration or conservative defaults.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+
+from services.intelligence_pipeline_v3.confidence.models import ConfidenceFeatures
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class ExtractionStageResult:
+    """Subset of extraction results relevant to confidence features.
+
+    This is an adapter interface — callers populate it from
+    the full extraction/specialist output.
+    """
+
+    entity_scores: list[float]
+    """Per-entity confidence scores from specialist extractor."""
+
+    relation_scores: list[float]
+    """Per-relation confidence scores."""
+
+    total_facts: int
+    """Total facts extracted."""
+
+    valid_numeric_facts: int
+    """Facts that passed deterministic parser validation."""
+
+    populated_fields: int
+    """Schema fields that have values."""
+
+    expected_fields: int
+    """Total expected schema fields for this document type."""
+
+
+@dataclass
+class ResolutionStageResult:
+    """Subset of resolution results relevant to confidence features."""
+
+    ambiguity_margins: list[float]
+    """Per-mention ambiguity margins (gap between top-2 candidates)."""
+
+
+@dataclass
+class EvidenceStageResult:
+    """Subset of evidence verification results relevant to confidence features."""
+
+    total_claims: int
+    """Total extracted claims/facts."""
+
+    supported_claims: int
+    """Claims backed by valid evidence spans."""
+
+
+@dataclass
+class SentimentStageResult:
+    """Subset of sentiment results relevant to confidence features."""
+
+    max_class_probabilities: list[float]
+    """Per-company maximum class probability after calibration."""
+
+    calibration_version: str
+    """Version of sentiment calibration artifact used."""
+
+
+@dataclass
+class AgreementStageResult:
+    """Cross-stage agreement analysis results."""
+
+    agreement_ratio: float
+    """Fraction of facts that agree across independent extraction paths."""
+
+    novelty_certainty: float
+    """Certainty of the novelty/duplicate classification (0-1)."""
+
+    hard_case_score: float
+    """Score indicating presence of known difficult patterns."""
+
+
+class ConfidenceFeatureExtractor:
+    """Extracts confidence features from upstream pipeline stage results.
+
+    Produces a normalized ConfidenceFeatures vector that can be passed
+    to the calibrator or used to determine conservative defaults.
+    """
+
+    def extract_features(
+        self,
+        extraction_result: ExtractionStageResult,
+        resolution_result: ResolutionStageResult,
+        evidence_result: EvidenceStageResult,
+        sentiment_result: SentimentStageResult,
+        agreement_result: AgreementStageResult | None = None,
+        document_type: str = "unknown",
+    ) -> ConfidenceFeatures:
+        """Compute confidence features from all upstream stage results.
+
+        Parameters
+        ----------
+        extraction_result
+            Entity/relation/fact extraction outputs with scores.
+        resolution_result
+            Symbol resolution outputs with ambiguity margins.
+        evidence_result
+            Evidence verification outputs with coverage stats.
+        sentiment_result
+            Sentiment classification outputs with calibrated probabilities.
+        agreement_result
+            Optional cross-stage agreement analysis. Defaults used if None.
+        document_type
+            Document type string for type-specific calibration.
+
+        Returns
+        -------
+        ConfidenceFeatures
+            Normalized feature vector ready for calibration.
+        """
+        # Entity span score: average of entity scores, or 0 if none
+        entity_span_score = (
+            sum(extraction_result.entity_scores) / len(extraction_result.entity_scores)
+            if extraction_result.entity_scores
+            else 0.0
+        )
+
+        # Alias resolution margin: average of per-mention margins
+        alias_resolution_margin = (
+            sum(resolution_result.ambiguity_margins)
+            / len(resolution_result.ambiguity_margins)
+            if resolution_result.ambiguity_margins
+            else 1.0  # No ambiguity if no mentions to resolve
+        )
+
+        # Numeric parser validity: fraction of valid numeric facts
+        numeric_parser_validity = (
+            extraction_result.valid_numeric_facts / extraction_result.total_facts
+            if extraction_result.total_facts > 0
+            else 1.0  # No numeric facts = no parser failures
+        )
+
+        # Evidence coverage: fraction of claims with valid evidence
+        evidence_coverage = (
+            evidence_result.supported_claims / evidence_result.total_claims
+            if evidence_result.total_claims > 0
+            else 0.0
+        )
+
+        # Relation score: average relation confidence
+        relation_score = (
+            sum(extraction_result.relation_scores)
+            / len(extraction_result.relation_scores)
+            if extraction_result.relation_scores
+            else 0.0
+        )
+
+        # Sentiment calibration confidence: average max class probability
+        sentiment_calibration_confidence = (
+            sum(sentiment_result.max_class_probabilities)
+            / len(sentiment_result.max_class_probabilities)
+            if sentiment_result.max_class_probabilities
+            else 0.5  # Neutral default when no sentiment data
+        )
+
+        # Document completeness: fraction of expected fields populated
+        document_completeness = (
+            extraction_result.populated_fields / extraction_result.expected_fields
+            if extraction_result.expected_fields > 0
+            else 0.0
+        )
+
+        # Cross-stage agreement features (use defaults if not provided)
+        if agreement_result is not None:
+            cross_stage_agreement = agreement_result.agreement_ratio
+            duplicate_novelty_certainty = agreement_result.novelty_certainty
+            known_hard_case_patterns = agreement_result.hard_case_score
+        else:
+            cross_stage_agreement = 0.5  # Neutral default
+            duplicate_novelty_certainty = 0.5
+            known_hard_case_patterns = 0.0
+
+        # Validate document type
+        valid_types = {
+            "news",
+            "filing",
+            "transcript",
+            "press_release",
+            "macro_event",
+            "unknown",
+        }
+        if document_type not in valid_types:
+            logger.warning(
+                "Unknown document_type '%s', defaulting to 'unknown'", document_type
+            )
+            document_type = "unknown"
+
+        return ConfidenceFeatures(
+            entity_span_score=_clamp(entity_span_score),
+            alias_resolution_margin=_clamp(alias_resolution_margin),
+            numeric_parser_validity=_clamp(numeric_parser_validity),
+            evidence_coverage=_clamp(evidence_coverage),
+            relation_score=_clamp(relation_score),
+            sentiment_calibration_confidence=_clamp(sentiment_calibration_confidence),
+            cross_stage_agreement=_clamp(cross_stage_agreement),
+            duplicate_novelty_certainty=_clamp(duplicate_novelty_certainty),
+            document_completeness=_clamp(document_completeness),
+            document_type=document_type,
+            known_hard_case_patterns=_clamp(known_hard_case_patterns),
+        )
+
+
+def _clamp(value: float, low: float = 0.0, high: float = 1.0) -> float:
+    """Clamp value to [low, high]."""
+    return max(low, min(high, value))

services/intelligence_pipeline_v3/confidence/models.py

@@ -0,0 +1,179 @@
+"""Pydantic models for confidence calibration pipeline.
+
+Defines feature vectors, calibration artifact metadata, and
+confidence results used throughout the confidence pipeline.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from typing import Literal
+
+from pydantic import BaseModel, Field, field_validator
+
+
+class ConfidenceFeatures(BaseModel):
+    """Feature vector for confidence estimation.
+
+    Each feature is a normalized float derived from upstream pipeline
+    stages: extraction, resolution, evidence verification, sentiment,
+    and cross-stage agreement analysis.
+    """
+
+    entity_span_score: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Best entity span confidence from specialist extractor.",
+    )
+    alias_resolution_margin: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Gap between top-2 alias candidates. 1.0 = unambiguous.",
+    )
+    numeric_parser_validity: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Fraction of numeric facts that passed parser validation.",
+    )
+    evidence_coverage: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Fraction of extracted facts backed by valid evidence spans.",
+    )
+    relation_score: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Average confidence of extracted relations.",
+    )
+    sentiment_calibration_confidence: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Calibrated sentiment model confidence (max class probability).",
+    )
+    cross_stage_agreement: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Agreement ratio between independently derived facts across stages.",
+    )
+    duplicate_novelty_certainty: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Certainty of the novelty/duplicate classification.",
+    )
+    document_completeness: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Fraction of expected schema fields that were populated.",
+    )
+    document_type: str = Field(
+        description="Document type (news, filing, transcript, press_release, macro_event).",
+    )
+    known_hard_case_patterns: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Score indicating presence of known hard patterns (multi-company, contradictions).",
+    )
+
+    @field_validator("document_type")
+    @classmethod
+    def document_type_valid(cls, v: str) -> str:
+        valid_types = {
+            "news",
+            "filing",
+            "transcript",
+            "press_release",
+            "macro_event",
+            "unknown",
+        }
+        if v not in valid_types:
+            raise ValueError(f"document_type must be one of {valid_types}, got '{v}'")
+        return v
+
+    def to_vector(self) -> list[float]:
+        """Convert features to a flat numeric vector for calibration models.
+
+        document_type is encoded as a categorical index.
+        """
+        type_map = {
+            "news": 0.0,
+            "filing": 0.2,
+            "transcript": 0.4,
+            "press_release": 0.6,
+            "macro_event": 0.8,
+            "unknown": 1.0,
+        }
+        return [
+            self.entity_span_score,
+            self.alias_resolution_margin,
+            self.numeric_parser_validity,
+            self.evidence_coverage,
+            self.relation_score,
+            self.sentiment_calibration_confidence,
+            self.cross_stage_agreement,
+            self.duplicate_novelty_certainty,
+            self.document_completeness,
+            type_map.get(self.document_type, 1.0),
+            self.known_hard_case_patterns,
+        ]
+
+
+class CalibrationArtifactMetadata(BaseModel):
+    """Metadata for a versioned calibration artifact.
+
+    Stored alongside the serialized calibrator to track provenance,
+    training conditions, and quality metrics.
+    """
+
+    version: str = Field(description="Artifact version string (e.g., 'v1.0.0').")
+    method: Literal["isotonic", "platt"] = Field(
+        description="Calibration method used."
+    )
+    training_count: int = Field(
+        ge=0, description="Number of samples used for training."
+    )
+    training_range: str = Field(
+        description="Date range of training data (e.g., '2024-01-01 to 2024-06-30')."
+    )
+    ece: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Expected Calibration Error on held-out data.",
+    )
+    brier_score: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Brier score on held-out data.",
+    )
+    created_at: datetime = Field(
+        default_factory=lambda: datetime.now(tz=timezone.utc),
+        description="When the artifact was created.",
+    )
+
+
+class ConfidenceResult(BaseModel):
+    """Final confidence output for an extraction record.
+
+    Contains the calibrated probability, feature breakdown, and
+    metadata about whether calibration was applied or defaults used.
+    """
+
+    probability: float = Field(
+        ge=0.0,
+        le=1.0,
+        description="Calibrated probability of extraction correctness.",
+    )
+    features_used: list[str] = Field(
+        description="Names of features that contributed to this confidence score.",
+    )
+    is_calibrated: bool = Field(
+        default=True,
+        description="Whether a trained calibrator was used (vs conservative default).",
+    )
+    under_calibrated: bool = Field(
+        default=False,
+        description="True if class has insufficient calibration data and conservative default was applied.",
+    )
+    calibration_version: str = Field(
+        default="uncalibrated",
+        description="Version of the calibration artifact used.",
+    )

services/intelligence_pipeline_v3/deprecation/__init__.py

@@ -0,0 +1,20 @@
+"""Legacy path deprecation tracking and cleanup management.
+
+Tracks deprecated components (VLLMClient, v2 prompts, provider branching),
+validates that all consumers have migrated, and provides safe removal
+gating. Removal only proceeds after all downstream consumers read v3.
+"""
+
+from services.intelligence_pipeline_v3.deprecation.tracker import (
+    DeprecationEntry,
+    DeprecationStatus,
+    DeprecationTracker,
+    MigrationReport,
+)
+
+__all__ = [
+    "DeprecationEntry",
+    "DeprecationStatus",
+    "DeprecationTracker",
+    "MigrationReport",
+]

services/intelligence_pipeline_v3/deprecation/tracker.py

@@ -0,0 +1,271 @@
+"""Deprecation tracker for legacy pipeline components.
+
+Manages the lifecycle of deprecated components: VLLMClient, v2 prompts,
+provider branching, 8000-char truncation, environment/model defaults,
+provider free-text fields, and the compatibility adapter.
+
+Removal only happens after all downstream consumers read v3 natively,
+validated by consumer audit.
+"""
+
+from __future__ import annotations
+
+import enum
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID, uuid4
+
+
+class DeprecationStatus(str, enum.Enum):
+    """Lifecycle status of a deprecated component."""
+
+    ACTIVE = "active"  # Still in use
+    DEPRECATED = "deprecated"  # Marked for removal, consumers migrating
+    MIGRATION_COMPLETE = "migration_complete"  # All consumers migrated
+    REMOVED = "removed"  # Code removed
+    ARCHIVED = "archived"  # Final reports preserved
+
+
+@dataclass
+class DeprecationEntry:
+    """A tracked deprecated component with migration status."""
+
+    entry_id: UUID
+    component_name: str
+    component_path: str  # File/module path
+    status: DeprecationStatus
+    deprecated_at: datetime
+    reason: str
+
+    # Consumer tracking
+    known_consumers: list[str] = field(default_factory=list)
+    migrated_consumers: list[str] = field(default_factory=list)
+
+    # Removal gates
+    removal_approved: bool = False
+    removal_approver: str = ""
+    removed_at: datetime | None = None
+
+    # Migration tracking
+    replacement: str = ""  # What replaces this component
+    migration_notes: str = ""
+
+    @classmethod
+    def create(
+        cls,
+        component_name: str,
+        component_path: str,
+        reason: str,
+        known_consumers: list[str] | None = None,
+        replacement: str = "",
+    ) -> DeprecationEntry:
+        return cls(
+            entry_id=uuid4(),
+            component_name=component_name,
+            component_path=component_path,
+            status=DeprecationStatus.DEPRECATED,
+            deprecated_at=datetime.now(timezone.utc),
+            reason=reason,
+            known_consumers=known_consumers or [],
+            replacement=replacement,
+        )
+
+    @property
+    def migration_progress(self) -> float:
+        """Fraction of consumers that have migrated (0.0-1.0)."""
+        if not self.known_consumers:
+            return 1.0
+        return len(self.migrated_consumers) / len(self.known_consumers)
+
+    @property
+    def all_consumers_migrated(self) -> bool:
+        """Whether all known consumers have migrated."""
+        return set(self.known_consumers) <= set(self.migrated_consumers)
+
+    def mark_consumer_migrated(self, consumer: str) -> None:
+        """Record that a consumer has migrated off this component."""
+        if consumer not in self.migrated_consumers:
+            self.migrated_consumers.append(consumer)
+        if self.all_consumers_migrated:
+            self.status = DeprecationStatus.MIGRATION_COMPLETE
+
+    def approve_removal(self, approver: str) -> bool:
+        """Approve removal. Only valid if all consumers migrated.
+
+        Returns False if removal cannot be approved.
+        """
+        if not self.all_consumers_migrated:
+            return False
+        self.removal_approved = True
+        self.removal_approver = approver
+        return True
+
+    def mark_removed(self) -> None:
+        """Record that the component has been removed from code."""
+        self.status = DeprecationStatus.REMOVED
+        self.removed_at = datetime.now(timezone.utc)
+
+    def archive(self) -> None:
+        """Archive after final migration reports preserved."""
+        self.status = DeprecationStatus.ARCHIVED
+
+
+# Default deprecation entries for the v3 migration
+DEFAULT_DEPRECATIONS: list[dict[str, Any]] = [
+    {
+        "component_name": "VLLMClient",
+        "component_path": "services/extractor/vllm_client.py",
+        "reason": "Replaced by OpenAICompatibleClient via inference gateway",
+        "known_consumers": [
+            "services/extractor/llm_factory.py",
+            "services/extractor/worker.py",
+        ],
+        "replacement": "services/shared/inference/clients/openai_compatible.py",
+    },
+    {
+        "component_name": "v2_extraction_prompt",
+        "component_path": "services/extractor/prompts.py",
+        "reason": "Monolithic prompt replaced by staged specialist extraction",
+        "known_consumers": [
+            "services/extractor/worker.py",
+        ],
+        "replacement": "services/intelligence_pipeline_v3/adjudication/",
+    },
+    {
+        "component_name": "provider_branching",
+        "component_path": "services/extractor/llm_factory.py",
+        "reason": "Duplicated if/else provider branching replaced by registry",
+        "known_consumers": [
+            "services/extractor/worker.py",
+            "services/recommendation/thesis_llm.py",
+        ],
+        "replacement": "services/shared/inference/registry.py",
+    },
+    {
+        "component_name": "8000_char_truncation",
+        "component_path": "services/extractor/prompts.py",
+        "reason": "Truncation replaced by sentence-aware segmenter",
+        "known_consumers": [
+            "services/extractor/prompts.py",
+        ],
+        "replacement": "services/intelligence_pipeline_v3/segmenter/",
+    },
+    {
+        "component_name": "compatibility_adapter",
+        "component_path": "services/intelligence_pipeline_v3/compatibility/",
+        "reason": "Temporary adapter removed after all consumers read v3 natively",
+        "known_consumers": [
+            "services/aggregation/worker.py",
+            "services/recommendation/",
+            "services/query_api/",
+        ],
+        "replacement": "Direct v3 intelligence records",
+    },
+]
+
+
+@dataclass
+class MigrationReport:
+    """Summary report of the deprecation/migration status."""
+
+    generated_at: datetime
+    total_components: int = 0
+    deprecated: int = 0
+    migration_complete: int = 0
+    removed: int = 0
+    blocked_removals: list[str] = field(default_factory=list)
+
+    @classmethod
+    def generate(cls, entries: list[DeprecationEntry]) -> MigrationReport:
+        report = cls(
+            generated_at=datetime.now(timezone.utc),
+            total_components=len(entries),
+        )
+        for entry in entries:
+            if entry.status == DeprecationStatus.DEPRECATED:
+                report.deprecated += 1
+                if not entry.all_consumers_migrated:
+                    remaining = set(entry.known_consumers) - set(
+                        entry.migrated_consumers
+                    )
+                    report.blocked_removals.append(
+                        f"{entry.component_name}: waiting on {list(remaining)}"
+                    )
+            elif entry.status == DeprecationStatus.MIGRATION_COMPLETE:
+                report.migration_complete += 1
+            elif entry.status in (
+                DeprecationStatus.REMOVED,
+                DeprecationStatus.ARCHIVED,
+            ):
+                report.removed += 1
+        return report
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "generated_at": self.generated_at.isoformat(),
+            "total_components": self.total_components,
+            "deprecated": self.deprecated,
+            "migration_complete": self.migration_complete,
+            "removed": self.removed,
+            "blocked_removals": self.blocked_removals,
+        }
+
+
+@dataclass
+class DeprecationTracker:
+    """Tracks all deprecated components and their migration status.
+
+    Enforces that removal only happens after all consumers migrate
+    and with explicit approval.
+    """
+
+    _entries: dict[str, DeprecationEntry] = field(default_factory=dict)
+
+    def add(self, entry: DeprecationEntry) -> None:
+        """Register a deprecated component."""
+        self._entries[entry.component_name] = entry
+
+    def get(self, component_name: str) -> DeprecationEntry | None:
+        return self._entries.get(component_name)
+
+    def mark_migrated(self, component_name: str, consumer: str) -> bool:
+        """Record a consumer migration. Returns False if component not found."""
+        entry = self._entries.get(component_name)
+        if entry is None:
+            return False
+        entry.mark_consumer_migrated(consumer)
+        return True
+
+    def can_remove(self, component_name: str) -> bool:
+        """Check if a component can be safely removed."""
+        entry = self._entries.get(component_name)
+        if entry is None:
+            return False
+        return entry.all_consumers_migrated and entry.removal_approved
+
+    def approve_removal(self, component_name: str, approver: str) -> bool:
+        """Approve removal of a component."""
+        entry = self._entries.get(component_name)
+        if entry is None:
+            return False
+        return entry.approve_removal(approver)
+
+    def generate_report(self) -> MigrationReport:
+        """Generate a migration status report."""
+        return MigrationReport.generate(list(self._entries.values()))
+
+    @property
+    def all_entries(self) -> list[DeprecationEntry]:
+        return list(self._entries.values())
+
+    @property
+    def pending_removals(self) -> list[DeprecationEntry]:
+        """Entries that are ready for removal (migrated + approved)."""
+        return [
+            e
+            for e in self._entries.values()
+            if e.all_consumers_migrated
+            and e.removal_approved
+            and e.status != DeprecationStatus.REMOVED
+        ]

services/intelligence_pipeline_v3/evaluation/__init__.py

@@ -0,0 +1 @@
+"""Evaluation metrics for Intelligence Pipeline v3."""

services/intelligence_pipeline_v3/evaluation/entity_metrics.py

@@ -0,0 +1,396 @@
+"""Entity and ticker precision, recall, F1, and ambiguity accuracy metrics.
+
+Implements evaluation metrics for entity extraction quality against a gold
+standard corpus. Supports both strict matching (exact span) and relaxed
+matching (overlapping span with same type), with per-type breakdowns.
+
+Validates: Requirements 16.3, 16.4
+"""
+from __future__ import annotations
+
+from enum import Enum
+from typing import Literal
+
+from pydantic import BaseModel, Field
+
+# ---------------------------------------------------------------------------
+# Domain Models
+# ---------------------------------------------------------------------------
+
+
+class MatchMode(str, Enum):
+    """Entity matching strategy."""
+
+    strict = "strict"
+    relaxed = "relaxed"
+
+
+class EntitySpan(BaseModel):
+    """A single entity mention with character offsets and type."""
+
+    text: str
+    entity_type: str
+    start_char: int
+    end_char: int
+    document_id: str = ""
+    canonical_id: str | None = None
+    is_ambiguous: bool = False
+
+    @property
+    def span(self) -> tuple[int, int]:
+        return (self.start_char, self.end_char)
+
+
+class TickerMention(BaseModel):
+    """A resolved ticker/company mention."""
+
+    text: str
+    ticker: str
+    start_char: int
+    end_char: int
+    document_id: str = ""
+    canonical_company_id: str | None = None
+    is_ambiguous: bool = False
+
+    @property
+    def span(self) -> tuple[int, int]:
+        return (self.start_char, self.end_char)
+
+
+# ---------------------------------------------------------------------------
+# Result Models
+# ---------------------------------------------------------------------------
+
+
+class PRF1(BaseModel):
+    """Precision, recall, F1 triple."""
+
+    precision: float = Field(ge=0.0, le=1.0)
+    recall: float = Field(ge=0.0, le=1.0)
+    f1: float = Field(ge=0.0, le=1.0)
+    support_predicted: int = Field(ge=0)
+    support_gold: int = Field(ge=0)
+
+
+class EntityMetricsResult(BaseModel):
+    """Full entity evaluation result with per-type breakdowns."""
+
+    match_mode: Literal["strict", "relaxed"]
+    overall: PRF1
+    per_type: dict[str, PRF1]
+
+
+class TickerMetricsResult(BaseModel):
+    """Ticker/company resolution evaluation result."""
+
+    match_mode: Literal["strict", "relaxed"]
+    overall: PRF1
+    per_type: dict[str, PRF1] = Field(
+        default_factory=dict,
+        description="Breakdown by canonical company or sector if available",
+    )
+
+
+class AmbiguityResult(BaseModel):
+    """Ambiguity detection accuracy."""
+
+    accuracy: float = Field(ge=0.0, le=1.0)
+    true_positives: int = Field(ge=0)
+    true_negatives: int = Field(ge=0)
+    false_positives: int = Field(ge=0)
+    false_negatives: int = Field(ge=0)
+    support: int = Field(ge=0)
+
+
+class EntityEvaluationReport(BaseModel):
+    """Complete entity evaluation report."""
+
+    entity_metrics: EntityMetricsResult
+    ticker_metrics: TickerMetricsResult
+    ambiguity_accuracy: AmbiguityResult
+    document_count: int = Field(ge=0)
+
+
+# ---------------------------------------------------------------------------
+# Matching Logic
+# ---------------------------------------------------------------------------
+
+
+def _spans_overlap(a: tuple[int, int], b: tuple[int, int]) -> bool:
+    """Return True if two character spans overlap."""
+    return a[0] < b[1] and b[0] < a[1]
+
+
+def _entity_matches_strict(pred: EntitySpan, gold: EntitySpan) -> bool:
+    """Strict match: exact span boundaries and same entity type."""
+    return (
+        pred.entity_type == gold.entity_type
+        and pred.start_char == gold.start_char
+        and pred.end_char == gold.end_char
+    )
+
+
+def _entity_matches_relaxed(pred: EntitySpan, gold: EntitySpan) -> bool:
+    """Relaxed match: overlapping span with same entity type."""
+    return pred.entity_type == gold.entity_type and _spans_overlap(
+        pred.span, gold.span
+    )
+
+
+def _ticker_matches_strict(pred: TickerMention, gold: TickerMention) -> bool:
+    """Strict match: exact span and same resolved ticker."""
+    return (
+        pred.ticker == gold.ticker
+        and pred.start_char == gold.start_char
+        and pred.end_char == gold.end_char
+    )
+
+
+def _ticker_matches_relaxed(pred: TickerMention, gold: TickerMention) -> bool:
+    """Relaxed match: overlapping span with same resolved ticker."""
+    return pred.ticker == gold.ticker and _spans_overlap(pred.span, gold.span)
+
+
+# ---------------------------------------------------------------------------
+# Core Metric Computation
+# ---------------------------------------------------------------------------
+
+
+def _compute_prf1(
+    predicted: list[EntitySpan] | list[TickerMention],
+    gold: list[EntitySpan] | list[TickerMention],
+    match_fn: object,
+) -> PRF1:
+    """Compute precision, recall, F1 using greedy bipartite matching.
+
+    Each predicted item can match at most one gold item and vice versa.
+    """
+    n_pred = len(predicted)
+    n_gold = len(gold)
+
+    if n_pred == 0 and n_gold == 0:
+        return PRF1(
+            precision=1.0,
+            recall=1.0,
+            f1=1.0,
+            support_predicted=0,
+            support_gold=0,
+        )
+
+    if n_pred == 0:
+        return PRF1(
+            precision=1.0,
+            recall=0.0,
+            f1=0.0,
+            support_predicted=0,
+            support_gold=n_gold,
+        )
+
+    if n_gold == 0:
+        return PRF1(
+            precision=0.0,
+            recall=1.0,
+            f1=0.0,
+            support_predicted=n_pred,
+            support_gold=0,
+        )
+
+    # Greedy matching: for each predicted, find first unmatched gold
+    matched_gold: set[int] = set()
+    true_positives = 0
+
+    for p in predicted:
+        for g_idx, g in enumerate(gold):
+            if g_idx in matched_gold:
+                continue
+            if match_fn(p, g):  # type: ignore[operator]
+                true_positives += 1
+                matched_gold.add(g_idx)
+                break
+
+    precision = true_positives / n_pred if n_pred > 0 else 0.0
+    recall = true_positives / n_gold if n_gold > 0 else 0.0
+
+    if precision + recall > 0:
+        f1 = 2 * precision * recall / (precision + recall)
+    else:
+        f1 = 0.0
+
+    return PRF1(
+        precision=precision,
+        recall=recall,
+        f1=f1,
+        support_predicted=n_pred,
+        support_gold=n_gold,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+
+def compute_entity_metrics(
+    predicted: list[EntitySpan],
+    gold: list[EntitySpan],
+    mode: MatchMode = MatchMode.strict,
+) -> EntityMetricsResult:
+    """Compute entity precision, recall, F1 with per-type breakdowns.
+
+    Args:
+        predicted: Predicted entity spans.
+        gold: Gold standard entity spans.
+        mode: Matching strategy (strict or relaxed).
+
+    Returns:
+        EntityMetricsResult with overall and per-type PRF1.
+    """
+    match_fn = _entity_matches_strict if mode == MatchMode.strict else _entity_matches_relaxed
+
+    # Overall
+    overall = _compute_prf1(predicted, gold, match_fn)
+
+    # Per-type breakdown
+    all_types = {e.entity_type for e in predicted} | {e.entity_type for e in gold}
+    per_type: dict[str, PRF1] = {}
+
+    for entity_type in sorted(all_types):
+        type_predicted = [e for e in predicted if e.entity_type == entity_type]
+        type_gold = [e for e in gold if e.entity_type == entity_type]
+        per_type[entity_type] = _compute_prf1(type_predicted, type_gold, match_fn)
+
+    return EntityMetricsResult(
+        match_mode=mode.value,
+        overall=overall,
+        per_type=per_type,
+    )
+
+
+def compute_ticker_metrics(
+    predicted: list[TickerMention],
+    gold: list[TickerMention],
+    mode: MatchMode = MatchMode.strict,
+) -> TickerMetricsResult:
+    """Compute ticker/company resolution precision, recall, F1.
+
+    Args:
+        predicted: Predicted ticker mentions with resolved tickers.
+        gold: Gold standard ticker mentions.
+        mode: Matching strategy (strict or relaxed).
+
+    Returns:
+        TickerMetricsResult with overall and optional per-ticker PRF1.
+    """
+    match_fn = _ticker_matches_strict if mode == MatchMode.strict else _ticker_matches_relaxed
+
+    overall = _compute_prf1(predicted, gold, match_fn)
+
+    # Per-ticker breakdown
+    all_tickers = {t.ticker for t in predicted} | {t.ticker for t in gold}
+    per_type: dict[str, PRF1] = {}
+
+    for ticker in sorted(all_tickers):
+        ticker_predicted = [t for t in predicted if t.ticker == ticker]
+        ticker_gold = [t for t in gold if t.ticker == ticker]
+        per_type[ticker] = _compute_prf1(ticker_predicted, ticker_gold, match_fn)
+
+    return TickerMetricsResult(
+        match_mode=mode.value,
+        overall=overall,
+        per_type=per_type,
+    )
+
+
+def compute_ambiguity_accuracy(
+    predicted: list[EntitySpan] | list[TickerMention],
+    gold: list[EntitySpan] | list[TickerMention],
+) -> AmbiguityResult:
+    """Compute ambiguity detection accuracy.
+
+    Measures how well the system identifies entities that require
+    adjudication (ambiguous entities). Uses the `is_ambiguous` flag
+    on each span/mention.
+
+    Entities are aligned by position (exact start_char, end_char match)
+    to compare ambiguity labels.
+
+    Args:
+        predicted: Predicted entities/tickers with ambiguity flags.
+        gold: Gold standard entities/tickers with ambiguity flags.
+
+    Returns:
+        AmbiguityResult with accuracy and confusion counts.
+    """
+    # Build a lookup from gold spans to ambiguity flag
+    gold_lookup: dict[tuple[int, int], bool] = {}
+    for g in gold:
+        gold_lookup[(g.start_char, g.end_char)] = g.is_ambiguous
+
+    tp = 0  # predicted ambiguous, gold ambiguous
+    tn = 0  # predicted not ambiguous, gold not ambiguous
+    fp = 0  # predicted ambiguous, gold not ambiguous
+    fn = 0  # predicted not ambiguous, gold ambiguous
+
+    matched_count = 0
+
+    for p in predicted:
+        key = (p.start_char, p.end_char)
+        if key in gold_lookup:
+            matched_count += 1
+            gold_ambiguous = gold_lookup[key]
+            pred_ambiguous = p.is_ambiguous
+
+            if pred_ambiguous and gold_ambiguous:
+                tp += 1
+            elif not pred_ambiguous and not gold_ambiguous:
+                tn += 1
+            elif pred_ambiguous and not gold_ambiguous:
+                fp += 1
+            else:
+                fn += 1
+
+    support = tp + tn + fp + fn
+    accuracy = (tp + tn) / support if support > 0 else 1.0
+
+    return AmbiguityResult(
+        accuracy=accuracy,
+        true_positives=tp,
+        true_negatives=tn,
+        false_positives=fp,
+        false_negatives=fn,
+        support=support,
+    )
+
+
+def evaluate_entities(
+    predicted_entities: list[EntitySpan],
+    gold_entities: list[EntitySpan],
+    predicted_tickers: list[TickerMention],
+    gold_tickers: list[TickerMention],
+    mode: MatchMode = MatchMode.strict,
+    document_count: int = 1,
+) -> EntityEvaluationReport:
+    """Run full entity evaluation producing a complete report.
+
+    Args:
+        predicted_entities: All predicted entity spans.
+        gold_entities: All gold standard entity spans.
+        predicted_tickers: All predicted ticker mentions.
+        gold_tickers: All gold standard ticker mentions.
+        mode: Matching strategy.
+        document_count: Number of documents evaluated.
+
+    Returns:
+        EntityEvaluationReport with entity metrics, ticker metrics,
+        and ambiguity accuracy.
+    """
+    entity_metrics = compute_entity_metrics(predicted_entities, gold_entities, mode)
+    ticker_metrics = compute_ticker_metrics(predicted_tickers, gold_tickers, mode)
+    ambiguity_accuracy = compute_ambiguity_accuracy(predicted_entities, gold_entities)
+
+    return EntityEvaluationReport(
+        entity_metrics=entity_metrics,
+        ticker_metrics=ticker_metrics,
+        ambiguity_accuracy=ambiguity_accuracy,
+        document_count=document_count,
+    )

services/intelligence_pipeline_v3/evaluation/event_metrics.py

@@ -0,0 +1,384 @@
+"""Event and relation macro/micro F1 evaluation metrics.
+
+Implements evaluation metrics for event classification and relation extraction
+quality against a gold standard corpus. Supports both macro-F1 (average across
+classes) and micro-F1 (global TP/FP/FN) with per-class breakdowns.
+
+Matching logic:
+- Events match if they share the same event_class AND have overlapping evidence
+  spans OR the same primary company.
+- Relations match if they share the same relation_type, source_id, and target_id.
+
+Validates: Requirements 16.3, 16.4
+"""
+from __future__ import annotations
+
+from pydantic import BaseModel, Field
+
+from services.intelligence_pipeline_v3.schemas.annotations import (
+    EventClass,
+    RelationType,
+)
+
+# ---------------------------------------------------------------------------
+# Input Models
+# ---------------------------------------------------------------------------
+
+
+class PredictedEvent(BaseModel):
+    """A predicted event for evaluation."""
+
+    event_class: EventClass
+    evidence_ids: list[str] = Field(default_factory=list)
+    primary_company_ids: list[str] = Field(default_factory=list)
+    confidence: float = Field(ge=0.0, le=1.0, default=1.0)
+
+
+class GoldEvent(BaseModel):
+    """A gold standard event for evaluation."""
+
+    event_class: EventClass
+    evidence_ids: list[str] = Field(default_factory=list)
+    primary_company_ids: list[str] = Field(default_factory=list)
+
+
+class PredictedRelation(BaseModel):
+    """A predicted relation for evaluation."""
+
+    relation_type: RelationType
+    source_id: str
+    target_id: str
+    confidence: float = Field(ge=0.0, le=1.0, default=1.0)
+
+
+class GoldRelation(BaseModel):
+    """A gold standard relation for evaluation."""
+
+    relation_type: RelationType
+    source_id: str
+    target_id: str
+
+
+# ---------------------------------------------------------------------------
+# Result Models
+# ---------------------------------------------------------------------------
+
+
+class PRF1(BaseModel):
+    """Precision, recall, F1 triple."""
+
+    precision: float = Field(ge=0.0, le=1.0)
+    recall: float = Field(ge=0.0, le=1.0)
+    f1: float = Field(ge=0.0, le=1.0)
+    support_predicted: int = Field(ge=0)
+    support_gold: int = Field(ge=0)
+
+
+class EventMetricsResult(BaseModel):
+    """Full event evaluation result with macro/micro F1 and per-class breakdown."""
+
+    macro_f1: float = Field(ge=0.0, le=1.0)
+    micro: PRF1
+    per_class: dict[str, PRF1]
+
+
+class RelationMetricsResult(BaseModel):
+    """Full relation evaluation result with macro/micro F1 and per-type breakdown."""
+
+    macro_f1: float = Field(ge=0.0, le=1.0)
+    micro: PRF1
+    per_type: dict[str, PRF1]
+
+
+class EventRelationEvaluationReport(BaseModel):
+    """Complete event and relation evaluation report."""
+
+    event_metrics: EventMetricsResult
+    relation_metrics: RelationMetricsResult
+    document_count: int = Field(ge=0)
+
+
+# ---------------------------------------------------------------------------
+# Matching Logic
+# ---------------------------------------------------------------------------
+
+
+def _events_match(pred: PredictedEvent, gold: GoldEvent) -> bool:
+    """Events match if same event_class AND overlapping evidence OR same primary company.
+
+    Overlap means at least one evidence_id in common, OR at least one
+    primary_company_id in common.
+    """
+    if pred.event_class != gold.event_class:
+        return False
+
+    # Check overlapping evidence spans
+    if pred.evidence_ids and gold.evidence_ids:
+        if set(pred.evidence_ids) & set(gold.evidence_ids):
+            return True
+
+    # Check same primary company
+    if pred.primary_company_ids and gold.primary_company_ids:
+        if set(pred.primary_company_ids) & set(gold.primary_company_ids):
+            return True
+
+    return False
+
+
+def _relations_match(pred: PredictedRelation, gold: GoldRelation) -> bool:
+    """Relations match if same type, source, and target."""
+    return (
+        pred.relation_type == gold.relation_type
+        and pred.source_id == gold.source_id
+        and pred.target_id == gold.target_id
+    )
+
+
+# ---------------------------------------------------------------------------
+# Core Metric Computation
+# ---------------------------------------------------------------------------
+
+
+def _compute_prf1_greedy(
+    predicted: list,
+    gold: list,
+    match_fn: object,
+) -> PRF1:
+    """Compute precision, recall, F1 using greedy bipartite matching.
+
+    Each predicted item can match at most one gold item and vice versa.
+    """
+    n_pred = len(predicted)
+    n_gold = len(gold)
+
+    if n_pred == 0 and n_gold == 0:
+        return PRF1(
+            precision=1.0, recall=1.0, f1=1.0,
+            support_predicted=0, support_gold=0,
+        )
+
+    if n_pred == 0:
+        return PRF1(
+            precision=1.0, recall=0.0, f1=0.0,
+            support_predicted=0, support_gold=n_gold,
+        )
+
+    if n_gold == 0:
+        return PRF1(
+            precision=0.0, recall=1.0, f1=0.0,
+            support_predicted=n_pred, support_gold=0,
+        )
+
+    matched_gold: set[int] = set()
+    true_positives = 0
+
+    for p in predicted:
+        for g_idx, g in enumerate(gold):
+            if g_idx in matched_gold:
+                continue
+            if match_fn(p, g):  # type: ignore[operator]
+                true_positives += 1
+                matched_gold.add(g_idx)
+                break
+
+    precision = true_positives / n_pred if n_pred > 0 else 0.0
+    recall = true_positives / n_gold if n_gold > 0 else 0.0
+
+    if precision + recall > 0:
+        f1 = 2 * precision * recall / (precision + recall)
+    else:
+        f1 = 0.0
+
+    return PRF1(
+        precision=precision,
+        recall=recall,
+        f1=f1,
+        support_predicted=n_pred,
+        support_gold=n_gold,
+    )
+
+
+def _compute_micro_prf1(
+    predicted: list,
+    gold: list,
+    match_fn: object,
+    class_key_pred: object,
+    class_key_gold: object,
+    all_classes: set[str],
+) -> PRF1:
+    """Compute micro-averaged PRF1 by summing TP/FP/FN across all classes."""
+    total_tp = 0
+    total_pred = 0
+    total_gold = 0
+
+    for cls in all_classes:
+        cls_predicted = [p for p in predicted if class_key_pred(p) == cls]
+        cls_gold = [g for g in gold if class_key_gold(g) == cls]
+
+        total_pred += len(cls_predicted)
+        total_gold += len(cls_gold)
+
+        # Greedy match within this class
+        matched_gold: set[int] = set()
+        for p in cls_predicted:
+            for g_idx, g in enumerate(cls_gold):
+                if g_idx in matched_gold:
+                    continue
+                if match_fn(p, g):  # type: ignore[operator]
+                    total_tp += 1
+                    matched_gold.add(g_idx)
+                    break
+
+    if total_pred == 0 and total_gold == 0:
+        return PRF1(
+            precision=1.0, recall=1.0, f1=1.0,
+            support_predicted=0, support_gold=0,
+        )
+
+    precision = total_tp / total_pred if total_pred > 0 else 0.0
+    recall = total_tp / total_gold if total_gold > 0 else 0.0
+
+    if precision + recall > 0:
+        f1 = 2 * precision * recall / (precision + recall)
+    else:
+        f1 = 0.0
+
+    return PRF1(
+        precision=precision,
+        recall=recall,
+        f1=f1,
+        support_predicted=total_pred,
+        support_gold=total_gold,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Public API — Events
+# ---------------------------------------------------------------------------
+
+
+def compute_event_metrics(
+    predicted: list[PredictedEvent],
+    gold: list[GoldEvent],
+) -> EventMetricsResult:
+    """Compute event macro-F1, micro-F1, and per-class F1.
+
+    Args:
+        predicted: Predicted events.
+        gold: Gold standard events.
+
+    Returns:
+        EventMetricsResult with macro, micro, and per-class breakdowns.
+    """
+    all_classes = {e.value for e in EventClass}
+
+    # Per-class breakdown
+    per_class: dict[str, PRF1] = {}
+    f1_scores: list[float] = []
+
+    for cls in sorted(all_classes):
+        cls_predicted = [p for p in predicted if p.event_class.value == cls]
+        cls_gold = [g for g in gold if g.event_class.value == cls]
+        prf1 = _compute_prf1_greedy(cls_predicted, cls_gold, _events_match)
+        per_class[cls] = prf1
+        f1_scores.append(prf1.f1)
+
+    # Macro-F1: average F1 across all event classes
+    macro_f1 = sum(f1_scores) / len(f1_scores) if f1_scores else 0.0
+
+    # Micro-F1: global TP/FP/FN
+    micro = _compute_micro_prf1(
+        predicted, gold, _events_match,
+        lambda p: p.event_class.value,
+        lambda g: g.event_class.value,
+        all_classes,
+    )
+
+    return EventMetricsResult(
+        macro_f1=macro_f1,
+        micro=micro,
+        per_class=per_class,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Public API — Relations
+# ---------------------------------------------------------------------------
+
+
+def compute_relation_metrics(
+    predicted: list[PredictedRelation],
+    gold: list[GoldRelation],
+) -> RelationMetricsResult:
+    """Compute relation macro-F1, micro-F1, and per-type F1.
+
+    Args:
+        predicted: Predicted relations.
+        gold: Gold standard relations.
+
+    Returns:
+        RelationMetricsResult with macro, micro, and per-type breakdowns.
+    """
+    all_types = {r.value for r in RelationType}
+
+    # Per-type breakdown
+    per_type: dict[str, PRF1] = {}
+    f1_scores: list[float] = []
+
+    for rtype in sorted(all_types):
+        type_predicted = [p for p in predicted if p.relation_type.value == rtype]
+        type_gold = [g for g in gold if g.relation_type.value == rtype]
+        prf1 = _compute_prf1_greedy(type_predicted, type_gold, _relations_match)
+        per_type[rtype] = prf1
+        f1_scores.append(prf1.f1)
+
+    # Macro-F1: average F1 across all relation types
+    macro_f1 = sum(f1_scores) / len(f1_scores) if f1_scores else 0.0
+
+    # Micro-F1: global TP/FP/FN
+    micro = _compute_micro_prf1(
+        predicted, gold, _relations_match,
+        lambda p: p.relation_type.value,
+        lambda g: g.relation_type.value,
+        all_types,
+    )
+
+    return RelationMetricsResult(
+        macro_f1=macro_f1,
+        micro=micro,
+        per_type=per_type,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Public API — Combined Report
+# ---------------------------------------------------------------------------
+
+
+def evaluate_events_and_relations(
+    predicted_events: list[PredictedEvent],
+    gold_events: list[GoldEvent],
+    predicted_relations: list[PredictedRelation],
+    gold_relations: list[GoldRelation],
+    document_count: int = 1,
+) -> EventRelationEvaluationReport:
+    """Run full event and relation evaluation producing a complete report.
+
+    Args:
+        predicted_events: All predicted events.
+        gold_events: All gold standard events.
+        predicted_relations: All predicted relations.
+        gold_relations: All gold standard relations.
+        document_count: Number of documents evaluated.
+
+    Returns:
+        EventRelationEvaluationReport with event metrics, relation metrics.
+    """
+    event_metrics = compute_event_metrics(predicted_events, gold_events)
+    relation_metrics = compute_relation_metrics(predicted_relations, gold_relations)
+
+    return EventRelationEvaluationReport(
+        event_metrics=event_metrics,
+        relation_metrics=relation_metrics,
+        document_count=document_count,
+    )