stonks-oracle/.kiro/specs/intelligence-pipeline-v3/design.md

Design Document

Overview

Intelligence Pipeline v3 replaces a monolithic "article to final trading-oriented JSON" request with a staged evidence and prediction architecture. The existing 9B model remains available, but its role changes from universal extractor and self-scorer to semantic adjudicator for the minority of documents that need broad language understanding.

The design intentionally chooses the best long-term architecture rather than the minimum code change:

• Generic OpenAI-compatible support is implemented as a capability-aware gateway, not another provider branch.
• Explicit facts, entities, numbers, and sentiment are produced by CPU-first specialist components.
• Novelty comes from retrieval and similarity.
• Confidence comes from empirical calibration.
• Impact and horizon come from a stock-specific model trained against realized outcomes.
• The existing 9B vLLM model handles ambiguity, causality, implication, and conflicts.
• Every field retains source evidence and model lineage.

Repository Review Findings

The following findings materially shaped this design:

Finding	Repository location	Consequence
The vLLM client receives a JSON Schema but sends only response_format: {"type": "json_object"}.	services/extractor/vllm_client.py:63-91	The server is not constraining generation to the actual schema.
vLLM extraction defaults to temperature 0.7.	services/shared/config.py:64-71, services/shared/config.py:284-291	Routine extraction is needlessly stochastic.
Unknown provider values silently fall back to Ollama.	services/extractor/llm_factory.py:1-6, services/extractor/llm_factory.py:47-67	Configuration mistakes can invoke the wrong endpoint without failing.
Long documents are truncated to the first 8,000 characters.	services/extractor/prompts.py:102-105	Filings, transcripts, and long articles can lose material facts.
The prompt asks one model for summary, entities, relevance, sentiment, impact, horizon, novelty, confidence, and evidence.	services/extractor/prompts.py:107-126	Extraction, reasoning, prediction, and self-evaluation are coupled.
The prompt supplies tracked tickers and invites inferred sector/theme exposure.	services/extractor/prompts.py:85-98	Explicit mentions and inferred exposure are mixed before evidence validation.
Persisted provider attribution is hardcoded to ollama, including failures.	services/extractor/worker.py:166-184, services/extractor/worker.py:227-244	Audit and model-performance attribution are incorrect for vLLM.
A single worker loop pops and processes one job at a time.	services/extractor/main.py:438-468, invocation near services/extractor/main.py:628	Application-level parallelism is constrained even if vLLM supports batching.
Runtime refresh mutates a client's private _config.	services/extractor/main.py:496-531	The protocol does not expose lifecycle or reconfiguration cleanly.
The thesis rewriter reimplements Ollama/vLLM branching.	services/recommendation/thesis_llm.py:87-200	Provider support is duplicated and will continue drifting.
Model defaults conflict across Python config, database migrations, Helm values, and the standalone vLLM deployment.	services/shared/config.py, infra/migrations, infra/helm/stonks-oracle/values.yaml, infra/kube-vllm/deployment.yaml	The repository cannot prove which model is canonical at runtime.
Model-produced novelty and confidence directly affect aggregation weight; model-produced impact is reused as sentiment strength and impact.	services/aggregation/scoring.py:436-529, services/aggregation/worker.py:430-463	Uncalibrated model self-scores can materially influence downstream signals.
A tracked Helm override contains plaintext production-like credentials.	infra/helm/stonks-oracle/values-live-math.yaml	Immediate rotation and history remediation are required before feature work ships.

The existing test suite around the LLM clients is useful. The focused provider tests passed after installing the declared dependencies plus the missing property-test dependency, but they encode current behavior and do not test true schema-constrained vLLM output.

Decision Summary

1. Add generic OpenAI-compatible support

Yes, but do not add an OpenAIClient beside VLLMClient and OllamaClient. Rename the concept:

• OllamaNativeClient for /api/chat and Ollama-specific controls.
• OpenAICompatibleClient for /v1/chat/completions and optionally /v1/responses after a separate compatibility gate.
• SpecialistHttpClient for typed non-generative endpoints.

vllm becomes a profile alias whose protocol is openai_chat. Hosted OpenAI, LM Studio, SGLang, LocalAI, or another compatible server can be represented by endpoint capabilities rather than new if provider == ... branches.

Use direct httpx requests in the generic layer. This keeps the wire payload explicit, permits provider-specific extra_body, simplifies redacted request auditing, and avoids binding every compatible server to one SDK's assumptions.

2. Retain the 9B model, but stop using it for every stage

The RTX 4070 Ti SUPER remains dedicated to one 9B-class vLLM deployment. This preserves the current semantic ceiling and current peak VRAM class. The 9B model is invoked only for ambiguous cases and receives a compressed evidence packet rather than the raw entire document and ticker universe.

3. Use a CPU-first fast path

Run the following on CPU nodes:

• GLiNER2 Large candidate for entities, event classes, relations, and schema-oriented extraction.
• FinBERT candidate for company-specific positive/negative/neutral probabilities.
• Deterministic parsers and the existing symbol registry for numeric facts and ticker identity.
• Compact embeddings plus fingerprints for novelty and deduplication.
• A calibrated tabular impact model for market direction, magnitude, and horizon.

NuExtract 1.5 Smol is retained as an evaluated optional stage for long-form or hierarchical extraction. It is not made a second always-resident GPU model because the intended deployment should preserve the 9B model's GPU footprint and because GLiNER2 plus deterministic parsing may already cover most literal extraction.

Architecture

flowchart TD
    A[Normalized document] --> B[Segmenter and offset map]
    B --> C[Deterministic candidates\ncompany aliases, tickers, numbers, dates]
    B --> D[GLiNER2 specialist\nentities, events, relations, facts]
    C --> E[Symbol resolver]
    D --> E
    E --> F[Evidence linker and verifier]
    F --> G[FinBERT per-company sentiment]
    F --> H[Novelty and dedup retrieval]
    G --> I[Confidence calibrator]
    H --> I
    I --> J{Fast-path acceptance?}
    J -->|yes| K[Approved evidence graph]
    J -->|no| L[9B Qwen adjudicator on vLLM]
    L --> M[Post-adjudication verifier]
    M --> K
    K --> N[Stock-specific impact model]
    N --> O[v3 intelligence records]
    O --> P[Compatibility adapter]
    P --> Q[Current aggregation and recommendation consumers]

Why this can be more intelligent without a larger footprint

A monolithic 9B model is broadly intelligent but is not necessarily the best estimator for every subproblem. The v3 design keeps that model for tasks requiring broad semantics while giving narrower jobs to components whose output can be calibrated and verified. The impact model adds information the language model does not have: observed historical market response. The result is not merely a smaller extractor; it is a system that combines textual reasoning with market-specific learned behavior.

Component Design

A. Inference Gateway

Package layout

services/shared/inference/
├── protocol.py
├── models.py
├── registry.py
├── router.py
├── capabilities.py
├── errors.py
├── redaction.py
└── clients/
    ├── ollama_native.py
    ├── openai_compatible.py
    └── specialist_http.py

Core types

@dataclass(frozen=True)
class ProviderCapabilities:
    chat_completions: bool
    responses_api: bool
    json_schema: bool
    json_object: bool
    seed: bool
    usage: bool
    max_completion_tokens: bool
    reasoning_toggle: bool
    model_listing: bool

@dataclass(frozen=True)
class InferenceTarget:
    endpoint_id: UUID
    deployment_id: UUID
    protocol: Literal["ollama_native", "openai_chat", "specialist_http"]
    base_url: str
    model: str
    capabilities: ProviderCapabilities
    auth_secret_ref: str | None
    extra_headers: Mapping[str, str]
    extra_body: Mapping[str, Any]

@dataclass
class StructuredGenerationRequest:
    messages: list[ChatMessage]
    json_schema: dict[str, Any] | None
    max_output_tokens: int
    temperature: float = 0.0
    seed: int | None = 0
    timeout_seconds: float = 120.0
    trace_id: str = ""

@dataclass
class InferenceResult:
    content: str
    parsed: dict[str, Any] | None
    target: InferenceTarget
    structured_mode: Literal["json_schema", "json_object", "prompt_only", "none"]
    latency_ms: int
    input_tokens: int | None
    output_tokens: int | None
    request_id: str | None
    repaired: bool
    retries: int

OpenAI-compatible structured output

The client chooses the strongest declared mode:

1. json_schema: send the actual schema and strict mode.
2. json_object: allow only if the deployment profile explicitly permits it.
3. prompt_only: allow only for experiments or legacy fallback.

For current vLLM versions, the gateway should support both standard response_format JSON Schema and a configurable vLLM structured_outputs extra body because deployed versions may differ. The endpoint profile records which wire form passed its capability probe.

Example standard payload:

{
  "model": "AxionML/Qwen3.5-9B-NVFP4",
  "messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}],
  "temperature": 0,
  "max_tokens": 1536,
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "adjudication_response",
      "strict": true,
      "schema": {}
    }
  }
}

The gateway validates the parsed response again locally. Wire constraints reduce malformed output; they do not replace semantic validation.

B. Endpoint Registry

New tables

CREATE TABLE inference_endpoints (
    id UUID PRIMARY KEY,
    name TEXT NOT NULL UNIQUE,
    protocol TEXT NOT NULL CHECK (protocol IN ('ollama_native','openai_chat','specialist_http')),
    base_url TEXT NOT NULL,
    auth_secret_ref TEXT,
    auth_scheme TEXT NOT NULL DEFAULT 'bearer',
    default_headers JSONB NOT NULL DEFAULT '{}',
    health_path TEXT,
    enabled BOOLEAN NOT NULL DEFAULT TRUE,
    revision INTEGER NOT NULL DEFAULT 1,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE model_deployments (
    id UUID PRIMARY KEY,
    endpoint_id UUID NOT NULL REFERENCES inference_endpoints(id),
    served_model_name TEXT NOT NULL,
    display_name TEXT NOT NULL,
    capabilities JSONB NOT NULL,
    context_window INTEGER,
    max_output_tokens INTEGER,
    quantization TEXT,
    runtime_metadata JSONB NOT NULL DEFAULT '{}',
    enabled BOOLEAN NOT NULL DEFAULT TRUE,
    revision INTEGER NOT NULL DEFAULT 1,
    UNIQUE(endpoint_id, served_model_name)
);

CREATE TABLE agent_stage_bindings (
    id UUID PRIMARY KEY,
    agent_id UUID NOT NULL REFERENCES ai_agents(id),
    stage TEXT NOT NULL,
    model_deployment_id UUID REFERENCES model_deployments(id),
    route_order INTEGER NOT NULL DEFAULT 0,
    routing_config JSONB NOT NULL DEFAULT '{}',
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    revision INTEGER NOT NULL DEFAULT 1,
    UNIQUE(agent_id, stage, route_order)
);

Authentication values are not stored in these tables. auth_secret_ref identifies a mounted secret or environment key understood by the deployment.

C. Document Segmenter

The segmenter replaces the 8,000-character prefix truncation.

Output

class DocumentChunk(BaseModel):
    chunk_id: str
    document_id: UUID
    document_type: str
    section_path: list[str]
    speaker: str | None
    start_char: int
    end_char: int
    text: str
    overlap_left: int
    overlap_right: int
    boilerplate_score: float

Suggested initial limits:

Document type	Target chunk size	Overlap	Notes
News / press release	700-1,000 tokens	100 tokens	Preserve paragraph boundaries.
Filing	900-1,300 tokens	150 tokens	Preserve headings and item sections.
Transcript	700-1,000 tokens	100 tokens	Preserve speaker turns.
Macro event	500-800 tokens	80 tokens	Favor compact event context.

The final values are benchmark parameters, not hard-coded assumptions.

D. Candidate Extraction and Symbol Resolution

Deterministic parsers generate high-precision candidates before specialist inference:

• Ticker tokens and exchange-qualified symbols.
• Currency and number expressions, including million, billion, ranges, percentages, basis points, and per-share amounts.
• Calendar and fiscal periods.
• Comparison cues such as up, down, beat, miss, raised, cut, above, and below.
• Company aliases from the symbol registry.

GLiNER2 receives focused schemas and returns spans for entities, event classes, relations, and structured facts. The resolver merges deterministic and specialist candidates using source offsets, aliases, and local context.

Explicit mentions and inferred exposures are different edge types:

Document --explicitly_mentions--> Company
Event --directly_affects--> Company
Event --inferred_exposure--> Company
Company --competes_with--> Company
Company --supplies--> Company

Only explicit and verified direct effects enter the primary company extraction. Inferred exposure continues through the existing interpolation/propagation architecture with separate confidence and provenance.

E. Sentiment Stage

FinBERT is run on company-linked evidence groups rather than the entire article. Each record contains:

class CompanySentiment(BaseModel):
    company_id: UUID
    evidence_ids: list[UUID]
    positive_probability: float
    negative_probability: float
    neutral_probability: float
    calibration_version: str
    model_deployment_id: UUID

Mixed sentiment is computed from multiple evidence groups and disagreement. It is not an unconstrained fourth softmax label.

F. Novelty Stage

Novelty becomes retrieval-based:

1. Compute an exact/near-duplicate fingerprint of normalized content.
2. Embed document chunks and canonical company-event representations.
3. Search a recent window in the vector index.
4. Calculate document novelty and event novelty from nearest-neighbor similarity, duplicate count, source timing, and event identity.
5. Store nearest matches for explainability.

A compact embedding model will be selected in the evaluation harness. The implementation must keep the embedding backend replaceable and must not entangle novelty scoring with the generative endpoint.

G. Confidence and Routing

Confidence features

• Entity span score.
• Alias-resolution margin between first and second candidate.
• Numeric parser validity.
• Evidence coverage.
• Relation score.
• Sentiment calibration confidence.
• Cross-stage agreement.
• Duplicate/novelty certainty.
• Document completeness.
• Document type and known hard-case patterns.

A calibration artifact maps these features to field-level correctness probabilities. Routing uses both calibrated confidence and hard rules.

Example adjudication triggers

UNRESOLVED_ALIAS
MULTIPLE_PRIMARY_COMPANIES
CONTRADICTORY_NUMERIC_FACTS
CONFLICTING_SENTIMENT
IMPLIED_CAUSAL_IMPACT
GUIDANCE_VS_CONSENSUS_REQUIRES_REASONING
MATERIAL_FIELD_MISSING
EVIDENCE_COVERAGE_BELOW_THRESHOLD
CALIBRATED_CONFIDENCE_BELOW_THRESHOLD
LONG_DOCUMENT_CROSS_CHUNK_RELATION

The desired initial target is 60-80 percent Fast_Path coverage after calibration. This is an evaluation target, not an assumed result.

H. Adjudication Packet

The 9B model receives only the information necessary to resolve a specific ambiguity:

class AdjudicationPacket(BaseModel):
    document_id: UUID
    document_type: str
    question_codes: list[str]
    candidate_companies: list[CompanyCandidate]
    candidate_events: list[EventCandidate]
    candidate_facts: list[FactCandidate]
    candidate_sentiments: list[CompanySentiment]
    evidence_spans: list[EvidenceSpan]
    relevant_chunks: list[DocumentChunk]
    required_decisions: list[str]

The adjudicator output does not contain final novelty, confidence, impact, or horizon. It resolves candidate identity, relationship, event interpretation, and supported qualitative direction. All output is evidence-linked and revalidated.

I. Impact and Horizon Model

This is the highest-impact stock-specific change.

Problem decomposition

• Text sentiment: What tone or directional implication is supported by the document?
• Event identity: What happened?
• Market impact: How has this kind of event historically affected this kind of security in this market regime?
• Horizon: Over what time window did the response usually manifest or decay?

The current model conflates these. v3 separates them.

Features

• Event class probability vector.
• Company-specific sentiment probability vector.
• Numeric magnitude and normalized surprise when consensus or prior value exists.
• Source credibility and historical source accuracy.
• Novelty and duplicate count.
• Evidence coverage and extraction uncertainty.
• Document type.
• Company sector, industry, market-cap bucket, liquidity, and beta.
• Pre-event volatility, volume regime, and broad market regime.
• Whether the event is direct, second-order, confirmed, quoted, or speculative.

Labels

Generate leakage-safe targets at defined event timestamps:

• Signed abnormal return relative to an approved benchmark.
• Absolute abnormal move.
• Abnormal volume.
• Direction labels for intraday, 1d, 7d, 30d, and 90d windows.
• Time to peak response and decay where data quality supports it.

Model family

Start with a deterministic event-weight baseline plus a CPU tabular learner such as gradient-boosted trees. Calibrate class probabilities out-of-time. The model artifact and feature pipeline are versioned independently.

The compatibility adapter may initially map expected signed magnitude to current impact_score and the most probable horizon to current impact_horizon, but richer distributions remain available to new consumers.

J. V3 Storage Schema

Suggested logical records:

class EvidenceSpan(BaseModel):
    id: UUID
    document_id: UUID
    chunk_id: str
    start_char: int
    end_char: int
    text: str
    checksum: str

class ExtractedEntity(BaseModel):
    id: UUID
    entity_type: str
    literal_text: str
    canonical_id: UUID | None
    evidence_id: UUID
    confidence: float
    derivation: str

class ExtractedFact(BaseModel):
    id: UUID
    fact_type: str
    subject_entity_id: UUID | None
    predicate: str
    literal_value: str
    normalized_value: dict | None
    period: dict | None
    evidence_ids: list[UUID]
    confidence: float
    derivation: str

class CompanySignalCandidate(BaseModel):
    company_id: UUID
    relevance_probability: float
    event_probabilities: dict[str, float]
    sentiment_probabilities: dict[str, float]
    direction_probabilities: dict[str, float]
    horizon_probabilities: dict[str, float]
    expected_magnitude: float | None
    evidence_ids: list[UUID]
    routing_reasons: list[str]
    adjudicated: bool

class StageLineage(BaseModel):
    stage: str
    endpoint_id: UUID | None
    deployment_id: UUID | None
    model_version: str | None
    schema_version: str
    calibration_version: str | None
    started_at: datetime
    duration_ms: int
    status: str

K. Compatibility Adapter

The adapter creates current records without discarding v3 provenance:

Current field	V3 source
summary	Deterministic template or optional 9B narrative generated from approved facts.
macro_themes	Approved event/theme classes.
novelty_score	Retrieval-derived event/document novelty.
confidence	Calibrated record correctness probability.
ticker	Canonical symbol registry resolution.
relevance	Calibrated direct-relevance probability.
sentiment	Company-specific calibrated distribution mapped to legacy enum.
impact_score	Approved impact-model magnitude mapped to legacy range.
impact_horizon	Most probable approved horizon.
catalyst_type	Versioned event taxonomy mapping.
evidence_spans	Exact source spans.

The adapter marks model_provider = 'hybrid' and stores complete stage lineage separately. No provider identity is hardcoded.

Deployment Design for RTX 4070 Ti SUPER Cluster

GPU deployment

One vLLM pod remains on the RTX 4070 Ti SUPER:

resources:
  limits:
    nvidia.com/gpu: 1
nodeSelector:
  accelerator: rtx-4070-ti-super
args:
  - --model
  - AxionML/Qwen3.5-9B-NVFP4
  - --served-model-name
  - stonks-adjudicator-9b
  - --max-model-len
  - "8192"
  - --max-num-seqs
  - "8"
  - --gpu-memory-utilization
  - "0.80"
  - --structured-outputs-config.backend
  - auto

Exact flags must match the pinned vLLM version. The deployment test must verify strict schema output before promotion.

CPU specialist deployment

replicas: 2
resources:
  requests:
    cpu: "2"
    memory: 4Gi
  limits:
    cpu: "6"
    memory: 10Gi

Initial pod contents:

• GLiNER2 Large.
• FinBERT.
• Tokenizers and deterministic parsers.
• Optional embedding model.

NuExtract 1.5 Smol should run as a separate benchmark or on-demand CPU deployment so its value can be measured independently.

Queue topology

extraction.incoming
    -> intelligence.router
        -> extraction.fast
        -> extraction.adjudication
        -> extraction.persist
        -> extraction.review

The router owns document state transitions. Workers use leases and idempotency keys so a retry cannot create duplicate intelligence records.

Concurrency

• Fast path: configurable worker pool, initially 4-8 concurrent documents per pod.
• Specialist API: micro-batching bounded by maximum wait time.
• Adjudicator: application semaphore aligned with vLLM max-num-seqs and measured KV-cache behavior.
• Persistence: independent bounded pool.

OpenAI-Compatible Support Details

Profiles

Profile	Protocol	Typical use
ollama	ollama_native	Existing Ollama endpoint.
vllm	openai_chat	Backward-compatible alias using vLLM capability profile.
openai	openai_chat	Hosted OpenAI endpoint with secret reference and egress policy.
openai_compatible	openai_chat	Any explicitly configured compatible server.
specialist	specialist_http	Typed GLiNER/FinBERT service.

Capability probes

A deployment activation test performs:

1. Health request.
2. Model listing if supported.
3. Minimal chat request.
4. Strict JSON Schema request.
5. Usage metadata check.
6. Seed/determinism check if declared.
7. Maximum output field compatibility check.

Probe results are stored with timestamp and software version. A failed capability cannot be enabled merely by selecting it in the UI.

Egress and data policy

Endpoint profiles include data-handling classification:

local_private
cluster_private
approved_external
forbidden_for_sensitive_docs

External endpoints are disabled by default. Routing to an approved external endpoint requires both an active binding and a document policy permitting egress.

Evaluation Strategy

Gold corpus

Build a minimum initial corpus of 1,000 human-reviewed documents, stratified across:

• News, filings, transcripts, and press releases.
• Single-company and multi-company stories.
• Earnings beats/misses and guidance changes.
• M&A, legal, regulatory, product, supply-chain, rating, management, and macro events.
• Explicit facts versus implied consequences.
• Short and long documents.
• Duplicate and recycled stories.

Compared systems

1. Current production path with current model and current prompt.
2. Current 9B model with corrected temperature and strict JSON Schema.
3. GLiNER2 + deterministic extraction.
4. GLiNER2 + deterministic extraction + FinBERT.
5. Optional NuExtract 1.5 Smol extraction path.
6. Full v3 fast path.
7. Full v3 with 9B adjudication.

This separation prevents architecture gains from being confused with a simple fix to the current vLLM request.

Metrics

• Company/ticker precision, recall, F1.
• Event macro-F1 and per-class F1.
• Numeric exact match with tolerance-aware normalization.
• Relation F1.
• Evidence support and offset validity.
• Company-specific sentiment macro-F1.
• Confidence ECE and Brier score.
• Unsupported-claim rate.
• JSON/schema failure rate.
• Fast-path coverage.
• Adjudication reason distribution.
• p50/p95 latency and documents per minute.
• CPU-seconds, GPU-seconds, tokens, and peak GPU memory per document.
• Impact-model direction accuracy, calibration, rank correlation, and out-of-time error by horizon.

Promotion sequence

1. Correct current vLLM schema constraints and temperature; establish baseline.
2. Run v3 offline replay.
3. Run v3 shadow mode.
4. Enable v3 for audit-only UI.
5. Canary compatibility outputs for a small percentage of non-trading downstream traffic.
6. Canary signal influence with automatic rollback.
7. Promote by document type and confidence tier.
8. Retire v2 only after a separately reviewed milestone.

Security Design

Immediate blocker

The repository contains plaintext production-like credentials in a tracked Helm values file. Treat them as compromised:

1. Rotate the database, object-store, Redis, broker, and market-data credentials.
2. Disable or replace old keys.
3. Remove secret values from tracked files.
4. Purge historical values using an approved Git history rewrite process.
5. Migrate to an external secret manager.
6. Add secret scanning to local hooks and CI.
7. Audit access logs for the affected credentials.

No values are reproduced in this specification.

Inference security

• Secrets are resolved only at runtime.
• Request logging records hashes and metadata, not authorization values.
• Raw source text is not logged at INFO level.
• External endpoint use is policy-gated.
• Stored raw prompts and responses use restricted object-store buckets and retention policies.
• Provider errors are normalized to avoid echoing secret-bearing response headers.

Testing Strategy

Unit tests

• Endpoint capability selection.
• Strict schema payload construction.
• Header and error redaction.
• Segment offset round trips.
• Numeric normalization.
• Alias resolution and ambiguity margins.
• Evidence linkage.
• Compatibility mappings.
• Confidence feature construction.
• Routing rules.

Property-based tests

• Every Evidence_Span round-trips to identical source text.
• Normalization never changes the literal stored value.
• Unknown providers always fail closed.
• Credentials never appear in serialized errors or logs.
• Compatibility mappings remain bounded in legacy field ranges.
• Reprocessing the same document and model versions is idempotent.
• Route decisions are deterministic for identical calibrated inputs.

Contract tests

• Ollama native endpoint.
• vLLM OpenAI-compatible endpoint.
• Mock hosted OpenAI-compatible endpoint.
• Specialist service schemas.
• Capability probe behavior across supported structured-output modes.

Integration tests

• Full document through fast path.
• Full document through adjudication path.
• Adjudicator outage with safe fast-path handling.
• Long filing crossing multiple chunks.
• Multi-company article with opposing sentiment.
• Duplicate story and novelty calculation.
• Rollback from v3 to v2.

Load tests

• CPU specialist batching.
• Queue backpressure.
• vLLM concurrent adjudication.
• Peak 4070 Ti SUPER memory.
• End-to-end throughput at representative article arrival rates.

Migration Plan

Phase 0: Security and baseline

Rotate secrets, add scanning, pin current runtime configuration, and benchmark the existing path.

Phase 1: Provider foundation

Add the Inference Gateway and registry. Convert current extractor, classifier, and thesis rewriter. Keep behavior otherwise equivalent.

Phase 2: Correct current generative extraction

Use strict schema-constrained vLLM output, temperature zero, accurate provider lineage, and bounded output sizes. This produces a fair current baseline.

Phase 3: V3 data and specialist shadow path

Add segmentation, v3 storage, deterministic extraction, GLiNER2, FinBERT, novelty, confidence, and audit UI. Persist shadow outputs only.

Phase 4: Adjudication and impact model

Add confidence routing, focused 9B adjudication, historical outcome features, deterministic impact baseline, and trained impact model.

Phase 5: Canary and promotion

Enable compatibility outputs by percentage and document type, then gradually allow v3 signals into aggregation.

Phase 6: Fine-tuning and cleanup

Fine-tune specialist models from reviewed cases, increase fast-path coverage, then remove deprecated v2 code in a separate change.

Risks and Mitigations

Risk	Mitigation
Specialist model misses implicit meaning.	Retain 9B adjudication with calibrated routing.
Complexity creates more failure modes.	Typed stage contracts, idempotent queues, stage-level metrics, and safe fallback.
Fast-path confidence is overestimated.	Held-out calibration, conservative thresholds, and shadow review.
Impact model learns leakage or regime artifacts.	Event-time feature snapshots, out-of-time validation, per-regime monitoring, and immutable predictions.
OpenAI-compatible servers differ subtly.	Capability probes and profile-specific wire settings, not optimistic assumptions.
A second model increases memory.	Keep specialists on CPU and make NuExtract optional/on-demand.
Existing downstream code assumes one model output.	Compatibility adapter and additive migrations.
Reviewer labels become inconsistent.	Annotation guide, double review for hard cases, and inter-annotator agreement tracking.