stonks-oracle/.kiro/specs/intelligence-pipeline-v3/requirements.md

Requirements Document

Introduction

Stonks Oracle currently asks a general-purpose generative model to perform entity discovery, ticker attribution, fact extraction, event classification, sentiment analysis, novelty estimation, confidence estimation, impact scoring, horizon selection, evidence quoting, and summarization in one response. That design is convenient, but it couples factual extraction to generative sampling and allows uncalibrated model self-assessments to influence signal weighting.

This specification introduces Intelligence Pipeline v3, a multi-stage, evidence-grounded inference system that preserves the current 9B model's reasoning ability for genuinely ambiguous documents while moving routine extraction, sentiment, novelty, confidence, and impact estimation into specialized and calibratable components. The target deployment retains the existing RTX 4070 Ti SUPER vLLM footprint and uses CPU-first specialist services for the fast path.

The specification also replaces provider-specific branching with a capability-aware inference gateway supporting Ollama native endpoints and generic OpenAI-compatible endpoints, including vLLM and hosted OpenAI-compatible services.

Goals

1. Match or exceed the current 9B pipeline's field-level accuracy and reasoning ceiling.
2. Reduce average GPU work per document without increasing peak GPU memory materially.
3. Make every extracted fact traceable to evidence in the source document.
4. Replace model-generated confidence, novelty, impact, and horizon values with calibrated or deterministic values.
5. Support generic OpenAI-compatible inference without adding another duplicated provider branch.
6. Establish a measurable benchmark, shadow rollout, and promotion process.
7. Preserve downstream compatibility while the v2 schema and database consumers are migrated.

Non-Goals

1. Replacing the existing recommendation, risk, or trading engines in one release.
2. Removing the current 9B model before the v3 pipeline passes promotion gates.
3. Treating backtest profit alone as proof of extraction correctness.
4. Sending credentials or proprietary documents to external providers by default.
5. Requiring a second GPU-resident generative model.

Glossary

• Inference_Gateway: Shared client and routing layer that invokes Ollama-native, OpenAI-compatible, and specialist inference endpoints through one typed interface.
• Endpoint_Profile: Persisted endpoint configuration containing protocol, URL, authentication reference, capabilities, and health settings.
• Model_Deployment: A model served by an Endpoint_Profile with declared capabilities and limits.
• Pipeline_Stage: One step in Intelligence Pipeline v3, such as segmentation, extraction, sentiment, verification, novelty, adjudication, or impact prediction.
• Fast_Path: CPU-first processing that completes without invoking the 9B generative model.
• Adjudication_Path: Processing that invokes the 9B model because evidence is ambiguous, contradictory, incomplete, or semantically complex.
• Evidence_Span: Exact source text plus stable character offsets and a chunk identifier.
• Candidate: A proposed entity, fact, event, sentiment, or relation before validation and calibration.
• Calibrated_Confidence: Probability-like confidence derived from validation data, not a number supplied by a generative model.
• Impact_Model: A lightweight supervised model that estimates signed market impact and horizon from extracted features and historical outcomes.
• Compatibility_Adapter: Mapper from v3 records to the current v2 document_intelligence and document_impact_records structures.
• Gold_Corpus: Human-reviewed documents and field-level labels used for acceptance testing.
• Shadow_Mode: Running v3 alongside the current pipeline without allowing v3 outputs to affect production decisions.

Requirements

Requirement 1: Secure Baseline and Credential Remediation

User Story: As an operator, I want repository and deployment credentials handled through secret stores, so that model-pipeline improvements do not ship on top of exposed credentials.

Acceptance Criteria

1. THE Team SHALL rotate every credential currently stored as plaintext in tracked repository files before deploying Intelligence Pipeline v3.
2. THE Repository SHALL remove plaintext database, object-store, Redis, broker, and market-data credentials from tracked Helm values and Git history.
3. THE Deployment SHALL reference credentials through Kubernetes Secrets populated by External Secrets, SOPS, Sealed Secrets, or an equivalent approved mechanism.
4. THE CI_Pipeline SHALL run secret scanning on pull requests and protected branches.
5. IF secret scanning detects a high-confidence credential, THEN THE CI_Pipeline SHALL fail before packaging or deployment.
6. THE Documentation SHALL record the rotation date and affected secret names without recording secret values.

Requirement 2: Capability-Aware Generic Inference Gateway

User Story: As a developer, I want one inference abstraction that supports Ollama and generic OpenAI-compatible services, so that endpoints can be changed without duplicating business logic.

Acceptance Criteria

1. THE Inference_Gateway SHALL support the protocols ollama_native, openai_chat, and specialist_http.
2. THE Inference_Gateway SHALL treat vllm as a backward-compatible profile alias for openai_chat, not as a separate client implementation.
3. WHEN an openai_chat request requires structured output and the endpoint declares json_schema support, THE Inference_Gateway SHALL send the complete supplied JSON Schema in strict structured-output mode.
4. WHEN an endpoint supports only JSON-object mode, THE Inference_Gateway SHALL use JSON-object mode only when that fallback is explicitly enabled for the Model_Deployment.
5. WHEN neither schema nor JSON-object constraints are supported, THE Inference_Gateway SHALL use prompt-only JSON generation only when explicitly enabled and SHALL mark the response as unconstrained.
6. IF a provider or protocol value is unknown, THEN THE Inference_Gateway SHALL fail closed with a configuration error and SHALL NOT silently route to Ollama.
7. THE Inference_Gateway SHALL support configurable base URL, request path, API-key secret reference, authorization scheme, additional headers, timeouts, retries, concurrency limit, and provider-specific extra request fields.
8. THE Inference_Gateway SHALL redact authentication values and configured sensitive headers from logs, traces, and stored request snapshots.
9. THE Inference_Gateway SHALL expose typed response metadata including endpoint ID, deployment ID, model name, protocol, request ID, latency, token usage, structured-output mode, retry count, and error category.
10. THE Inference_Gateway SHALL provide health and capability probes and cache their results with a bounded TTL.
11. WHEN endpoint capabilities are changed or a probe fails, THE Router SHALL invalidate the cached capability record before the next invocation.
12. THE Existing thesis rewriter, event classifier, and document extractor SHALL use the same Inference_Gateway rather than implementing separate Ollama/vLLM branches.

Requirement 3: Canonical Endpoint and Model Registry

User Story: As an operator, I want the database and UI to identify exactly which endpoint and model serve each stage, so that environment, migration, Helm, and runtime defaults cannot drift silently.

Acceptance Criteria

1. THE Database SHALL store inference_endpoints, model_deployments, and agent_stage_bindings as canonical runtime records.
2. EACH Inference_Endpoint SHALL include name, protocol, base URL, authentication secret reference, health path, default headers, enabled state, and timestamps.
3. EACH Model_Deployment SHALL include endpoint ID, served model name, display name, capabilities, context limit, output limit, quantization, structured-output modes, and enabled state.
4. EACH Agent_Stage_Binding SHALL map an agent and pipeline stage to one or more ordered Model_Deployments plus routing configuration.
5. WHEN runtime configuration is resolved, THE Service SHALL record the exact endpoint, deployment, and binding revision used.
6. THE API SHALL validate endpoint URLs, protocol values, capability declarations, and model names before activation.
7. THE UI SHALL use controlled protocol and endpoint selections rather than an unrestricted provider text field.
8. THE Migration SHALL translate existing ollama and vllm agent settings into Endpoint_Profile and Model_Deployment records without breaking active agents.
9. THE Application SHALL have one documented fallback configuration source; conflicting model defaults in code, migrations, and Helm SHALL be removed.

Requirement 4: Document Segmentation and Source Preservation

User Story: As an analyst, I want long articles, filings, and transcripts processed without destructive truncation, so that material facts near the end of a document are not lost.

Acceptance Criteria

1. THE Pipeline SHALL preserve the full normalized source document and SHALL NOT truncate it to a fixed character prefix for extraction.
2. THE Segmenter SHALL create sentence-aware chunks with stable chunk IDs, source character offsets, and configurable overlap.
3. THE Segmenter SHALL use document-type-specific chunk sizes for articles, filings, transcripts, and press releases.
4. THE Segmenter SHALL preserve headings, speaker labels, table-derived text markers, and section boundaries when present.
5. WHEN duplicate or boilerplate sections are detected, THE Segmenter SHALL mark them without deleting the only occurrence of a fact.
6. THE Pipeline SHALL retain a mapping from every downstream Evidence_Span to the original document offsets.
7. IF a document cannot be decoded or segmented, THEN THE Pipeline SHALL mark the document as a typed preprocessing failure and SHALL NOT fabricate an empty extraction.

Requirement 5: Deterministic Candidate Generation and Ticker Resolution

User Story: As a signal consumer, I want explicit companies and numeric facts resolved deterministically where possible, so that a language model is not asked to invent identifiers or parse trivial values.

Acceptance Criteria

1. THE Candidate_Generator SHALL detect explicit ticker symbols, company names, aliases, executives, products, currencies, percentages, dates, ranges, EPS values, revenue values, guidance values, and common financial ratios.
2. THE Symbol_Resolver SHALL use the existing company and symbol registry as the source of truth for ticker identity.
3. THE Pipeline SHALL distinguish explicit company mentions from inferred exposure relationships.
4. THE Pipeline SHALL NOT pass the entire tracked-ticker universe to a generative prompt.
5. WHEN multiple companies match an alias, THE Symbol_Resolver SHALL return ranked candidates and SHALL require contextual disambiguation or adjudication.
6. WHEN a ticker is not present in the symbol registry, THE Pipeline SHALL preserve the literal mention as unresolved rather than inventing a registered ticker.
7. THE Numeric_Normalizer SHALL retain both literal source text and normalized values, currencies, units, periods, and ranges.
8. THE Pipeline SHALL reject normalized numeric facts whose value cannot be traced to an Evidence_Span.

Requirement 6: Specialist Extraction Service

User Story: As an operator, I want routine entity, event, relation, and fact extraction to run on CPU-first specialist models, so that GPU capacity is reserved for difficult reasoning.

Acceptance Criteria

1. THE Specialist_Service SHALL expose batched APIs for entity extraction, schema extraction, relation extraction, and text classification.
2. THE Initial specialist extractor SHALL support company, person, product, event, financial metric, date, percentage, currency, and relationship schemas.
3. THE Specialist_Service SHALL return character spans and per-candidate scores for every extracted item.
4. THE Specialist_Service SHALL run without requiring the RTX 4070 Ti SUPER.
5. THE Initial deployment SHALL evaluate GLiNER2 Large as the primary unified extraction and classification model.
6. THE Benchmark SHALL evaluate NuExtract 1.5 Smol as an optional long-form or hierarchical fact-extraction stage, but it SHALL NOT become an always-resident GPU model without passing incremental-value and resource gates.
7. THE Specialist_Service SHALL support model version pinning, warm-up, health checks, bounded batching, and graceful degradation.
8. WHEN specialist inference fails, THE Router SHALL either retry according to policy or route to adjudication; it SHALL record the failure and SHALL NOT silently substitute default facts.
9. THE Specialist_Service SHALL expose model and schema versions in every response.

Requirement 7: Company-Specific Financial Sentiment

User Story: As an analyst, I want sentiment tied to each company and supporting evidence, so that a positive statement about one firm is not applied to every company in the article.

Acceptance Criteria

1. THE Sentiment_Stage SHALL score evidence sentences or evidence groups associated with each resolved company.
2. THE Initial sentiment classifier SHALL evaluate FinBERT as the baseline financial-domain model.
3. THE Sentiment_Stage SHALL return positive, negative, and neutral probabilities rather than only a discrete label.
4. THE Pipeline SHALL derive mixed sentiment from conflicting supported evidence, not from an unconstrained model label.
5. WHEN an article mentions competitors with opposing effects, THE Pipeline SHALL produce separate company-specific sentiment records.
6. THE Sentiment_Stage SHALL preserve the evidence IDs used for each probability distribution.
7. THE Production model SHALL be calibrated on the Gold_Corpus before its probabilities are treated as confidence values.

Requirement 8: Evidence Verification and Grounding

User Story: As an auditor, I want every material claim verified against source evidence, so that generated summaries and signals cannot rely on unsupported assertions.

Acceptance Criteria

1. EVERY material company fact, event, amount, direction, and relationship SHALL reference one or more Evidence_Spans.
2. THE Verifier SHALL check span validity, source offsets, entity association, and schema compatibility.
3. THE Benchmark SHALL evaluate a compact entailment verifier for claims that require semantic validation beyond exact matching.
4. IF a candidate conflicts with its evidence, THEN THE Pipeline SHALL reject it or route the conflict to adjudication.
5. THE Pipeline SHALL calculate evidence coverage as the proportion of required fields supported by valid spans.
6. THE Pipeline SHALL store rejected candidates and rejection reasons for audit and active learning.
7. JSON repair SHALL NOT transform an unsupported or truncated generative answer into a valid production extraction without marking it as repaired and revalidating every material field.

Requirement 9: Deterministic Novelty and Duplicate Detection

User Story: As a signal consumer, I want novelty based on comparison with recent information, so that a model's subjective novelty guess does not amplify repeated news.

Acceptance Criteria

1. THE Novelty_Stage SHALL compare each document and material event against a configurable recent-history window.
2. THE Novelty_Stage SHALL combine exact/near-duplicate fingerprints with compact semantic embeddings.
3. THE Pipeline SHALL calculate novelty separately for document-level content and company-event content.
4. THE Novelty_Stage SHALL return nearest matching document or event IDs plus similarity scores.
5. THE Pipeline SHALL derive novelty_score from the similarity distribution and duplicate count using a versioned deterministic formula or calibrated model.
6. A generative model SHALL NOT provide the authoritative novelty value used by aggregation.
7. WHEN novelty cannot be calculated because history is unavailable, THE Pipeline SHALL use a conservative versioned default and mark the reason.

Requirement 10: Calibrated Extraction Confidence

User Story: As a downstream scorer, I want confidence to reflect observed correctness, so that the system does not trust a model merely because it reports confidence in itself.

Acceptance Criteria

1. THE Pipeline SHALL calculate field-level and record-level confidence from specialist scores, symbol resolution, evidence validation, schema completeness, model agreement, and historical calibration.
2. A generative model's self-reported confidence SHALL NOT be used as authoritative extraction confidence.
3. THE Calibration_Process SHALL evaluate isotonic, Platt, or equivalent calibration methods on held-out Gold_Corpus data.
4. THE Pipeline SHALL report Expected Calibration Error and Brier score for probability-bearing stages.
5. THE Router SHALL use calibrated uncertainty and explicit conflict rules to choose Fast_Path or Adjudication_Path.
6. THE Pipeline SHALL retain stage-level confidence components for explainability.
7. WHEN calibration data is insufficient for a class, THE Pipeline SHALL use conservative thresholds and mark the class as under-calibrated.

Requirement 11: 9B Generative Adjudicator

User Story: As an analyst, I want the current reasoning capability retained for hard documents, so that specialization does not reduce intelligence on nuanced cases.

Acceptance Criteria

1. THE Adjudicator SHALL initially use the existing 9B-class model served by vLLM on the RTX 4070 Ti SUPER.
2. THE Adjudicator SHALL receive selected source chunks, Evidence_Spans, candidate facts, candidate probabilities, conflicts, and a precise adjudication question rather than the entire tracked ticker list.
3. THE Adjudicator SHALL use strict JSON Schema constrained output when supported by the endpoint.
4. THE Adjudicator SHALL use deterministic generation settings appropriate for extraction, including a production default temperature of zero unless a benchmark proves a different value superior.
5. THE Adjudicator SHALL NOT be asked to provide authoritative novelty, confidence, or impact values.
6. THE Router SHALL invoke adjudication for unresolved entity aliases, contradictory evidence, multi-company causal relationships, implied consequences, complex guidance, materially incomplete fast-path results, or low calibrated confidence.
7. THE Adjudicator SHALL return field-level decisions, evidence references, and decision reasons.
8. IF adjudication output references evidence not supplied to it, THEN THE Verifier SHALL reject the unsupported field.
9. THE Adjudicator SHALL remain optional for thesis prose; deterministic signal records SHALL not depend on prose generation succeeding.
10. THE Peak GPU memory budget SHALL not exceed the measured current 9B deployment baseline by more than 5 percent unless explicitly approved.

Requirement 12: Stock-Specific Impact and Horizon Model

User Story: As a trader, I want impact and horizon estimated from historical market behavior rather than language-model intuition, so that signals are tied to observed outcomes.

Acceptance Criteria

1. THE Pipeline SHALL separate textual sentiment from expected market impact.
2. THE Impact_Model SHALL consume versioned features including event type probabilities, sentiment probabilities, magnitude, surprise where available, source history, novelty, company attributes, market regime, pre-event volatility, and evidence quality.
3. THE Training_Pipeline SHALL create leakage-safe labels from abnormal returns and volume responses over configured horizons.
4. THE Initial model family SHALL be a CPU-efficient calibrated tabular model and SHALL include a transparent deterministic baseline.
5. THE Impact_Model SHALL output signed direction probabilities, expected magnitude, horizon probabilities, and model uncertainty.
6. THE Production model SHALL be evaluated out-of-time and by event type, sector, market-cap bucket, and source.
7. THE Pipeline SHALL preserve existing downstream fields through a Compatibility_Adapter while storing richer probability distributions in v3 tables.
8. IF no trained Impact_Model is approved, THEN THE Pipeline SHALL use the deterministic baseline and SHALL NOT fall back to a generative model's impact score.
9. THE Outcome_Evaluator SHALL feed realized outcomes back into model monitoring and retraining datasets without mutating historical predictions.
10. THE Pipeline SHALL version feature definitions, training data ranges, model artifacts, thresholds, and calibration artifacts.

Requirement 13: Versioned Intelligence Schema and Provenance

User Story: As a developer, I want a richer schema with field-level provenance, so that downstream consumers can distinguish facts, probabilities, decisions, and generated prose.

Acceptance Criteria

1. THE Database SHALL store v3 entities, facts, evidence spans, company signal candidates, stage runs, adjudication decisions, and model lineage in normalized or well-defined JSONB-backed tables.
2. EVERY v3 field SHALL identify whether it is deterministic, specialist-derived, adjudicated, calibrated, or compatibility-derived.
3. EVERY stage run SHALL record input references, output references, model versions, endpoint identity, duration, error state, and trace ID.
4. THE Compatibility_Adapter SHALL map approved v3 outputs to existing v2 persistence records during migration.
5. THE Compatibility_Adapter SHALL identify its own version and SHALL not overwrite original v3 probabilities.
6. THE persisted model_provider and model lineage SHALL reflect the actual route used and SHALL not be hardcoded to Ollama.
7. THE Pipeline SHALL retain raw model output only in approved object storage with configured retention and access controls.

Requirement 14: Parallelism, Queues, and Resource Isolation

User Story: As an operator, I want parallel throughput without saturating the GPU or blocking unrelated stages, so that the cluster remains responsive.

Acceptance Criteria

1. THE Extractor SHALL support multiple in-flight documents using bounded asynchronous workers rather than a single unbounded sequential loop.
2. THE Fast_Path and Adjudication_Path SHALL have separate queue or concurrency controls.
3. THE Specialist_Service SHALL support dynamic batching within configured latency limits.
4. THE Adjudicator SHALL enforce a GPU-safe concurrency semaphore coordinated with vLLM limits.
5. THE Router SHALL apply backpressure when either path exceeds its queue-depth or latency thresholds.
6. THE Deployment SHALL assign specialist workloads to CPU nodes and the 9B vLLM workload to the RTX 4070 Ti SUPER node by default.
7. THE System SHALL expose queue depth, service time, batch size, GPU memory, GPU utilization, fast-path rate, and adjudication rate.
8. WHEN the adjudicator is unavailable, THE Pipeline SHALL continue only for documents meeting a conservative fast-path acceptance threshold; all others SHALL remain queued or fail safely.

Requirement 15: Observability, Audit, and Explainability

User Story: As an operator and analyst, I want to understand why a document produced a signal and which component made each decision.

Acceptance Criteria

1. THE Pipeline SHALL emit one distributed trace covering preprocessing, specialist stages, routing, adjudication, impact prediction, and persistence.
2. THE Metrics SHALL include field validity, evidence coverage, entity resolution rate, sentiment agreement, calibration metrics, fast-path coverage, adjudication causes, schema failure rate, latency percentiles, token usage, and GPU-seconds per document.
3. THE Audit API SHALL return model lineage and evidence for a document, company, and generated signal.
4. THE UI SHALL distinguish observed facts, inferred exposure, sentiment, predicted impact, and generated narrative.
5. THE Pipeline SHALL store routing reasons as structured codes rather than log-only text.
6. THE Pipeline SHALL allow a reviewer to mark a field correct, incorrect, unsupported, or ambiguous and add a corrected value.
7. Reviewer corrections SHALL be immutable audit events and SHALL feed the active-learning dataset only through an approved export process.

Requirement 16: Benchmark, Shadow Mode, and Promotion Gates

User Story: As an owner, I want the new architecture proven against the current system before it affects trades, so that complexity is justified by measured improvement.

Acceptance Criteria

1. THE Team SHALL create a versioned Gold_Corpus covering articles, filings, press releases, transcripts, macro news, multi-company stories, contradictory reports, and long documents.
2. THE Evaluation_Harness SHALL run the current pipeline and every proposed v3 configuration on identical inputs.
3. THE Evaluation SHALL report field precision, recall, F1, exact-match accuracy, evidence support rate, ticker-resolution accuracy, event macro-F1, sentiment macro-F1, calibration, latency, throughput, CPU use, GPU use, and cost.
4. THE Evaluation SHALL report results by document type, event class, source, sector, and difficulty bucket.
5. THE Initial promotion gate SHALL require no statistically meaningful regression in any safety-critical field and measurable improvement in at least one of evidence support, calibration, schema validity, or resource efficiency.
6. THE Initial production target SHALL achieve at least 60 percent Fast_Path coverage on the representative corpus while meeting accuracy gates.
7. THE GPU-seconds per accepted document SHALL improve by at least 2x relative to the current 9B-every-document baseline before full promotion.
8. THE v3 pipeline SHALL run in Shadow_Mode for a configurable period and minimum document count before it may influence aggregation.
9. THE Promotion process SHALL support canary percentages, automatic rollback thresholds, and one-click reversion to the current pipeline.
10. Backtest or paper-trading performance SHALL be reported separately from extraction correctness and SHALL not override failed correctness gates.

Requirement 17: Active Learning and Specialist Fine-Tuning

User Story: As a model owner, I want difficult and corrected examples to improve the specialist path over time, so that fewer documents require the 9B adjudicator.

Acceptance Criteria

1. THE Active_Learning_Exporter SHALL select low-confidence, conflicting, adjudicated, and reviewer-corrected examples without exporting secrets or unauthorized content.
2. THE Export format SHALL retain source text, spans, schema labels, relations, adjudicator decisions, reviewer corrections, and provenance.
3. THE Training_Pipeline SHALL support fine-tuning the selected specialist extractor on the Stonks Oracle schema.
4. EACH trained artifact SHALL be evaluated against a frozen holdout and the current production artifact.
5. A specialist model SHALL not be promoted solely because it reduces adjudication rate; it SHALL also pass field-level correctness and calibration gates.
6. THE Registry SHALL retain model cards containing training range, dataset version, intended use, limitations, and evaluation results.

Requirement 18: Backward-Compatible Rollout

User Story: As a maintainer, I want to ship the new pipeline incrementally, so that existing APIs and downstream services continue operating during migration.

Acceptance Criteria

1. THE Current v2 extractor SHALL remain available behind a feature flag until v3 completes shadow and canary promotion.
2. THE Compatibility_Adapter SHALL produce the fields required by aggregation, recommendation, validation, reporting, and API consumers.
3. THE Database migration SHALL be additive before any destructive column or table change.
4. THE Deployment SHALL permit per-agent, per-document-type, and percentage-based routing between v2 and v3.
5. WHEN rollback is triggered, THE System SHALL route new work to v2 without deleting v3 audit data.
6. THE Team SHALL remove deprecated provider branches, v2 prompt logic, and compatibility mappings only in a separately approved cleanup milestone.