feat: Intelligence Pipeline v3 — full implementation
Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.
This commit is contained in:
Celes Renata
@@ -0,0 +1 @@
|
||||
{"specId": "ce34e647-8d91-4295-a3c0-7b001abccdee", "workflowType": "requirements-first", "specType": "feature"}
|
||||
@@ -0,0 +1,69 @@
|
||||
# Stonks Oracle Intelligence Architecture Review
|
||||
|
||||
## Recommendation
|
||||
|
||||
Add generic OpenAI-compatible support, but implement it as a protocol/capability layer rather than a third vendor-specific branch. Keep Ollama native support. Convert the existing vLLM path into an OpenAI-compatible endpoint profile.
|
||||
|
||||
For the RTX 4070 Ti SUPER cluster, do not replace the current 9B model with one smaller all-purpose model. Retain the 9B model as a focused adjudicator and split routine work into CPU-first specialist stages:
|
||||
|
||||
1. Deterministic parsing and symbol-registry resolution.
|
||||
2. GLiNER2 Large for entities, event classes, relations, and evidence spans.
|
||||
3. FinBERT for company-specific financial sentiment probabilities.
|
||||
4. Retrieval-based novelty and duplicate detection.
|
||||
5. Calibrated confidence from observed field correctness.
|
||||
6. A stock-specific tabular model trained on realized abnormal returns for impact and horizon.
|
||||
7. The existing 9B Qwen-class model for ambiguous, causal, multi-company, or implied reasoning.
|
||||
|
||||
This preserves the current reasoning ceiling, reduces average GPU inference, improves evidence fidelity, and adds stock-specific intelligence that a general language model cannot obtain from article text alone.
|
||||
|
||||
## Option Review
|
||||
|
||||
| Option | Best use | Weakness | Production role |
|
||||
|---|---|---|---|
|
||||
| Current Qwen3.5-class 9B monolith | Broad zero-shot semantics and hard reasoning | Expensive per document; stochastic; self-scores confidence/novelty/impact; weak calibration | Keep as adjudicator, not universal extractor |
|
||||
| Qwen3.5 4B | Smaller generalist | Lower reasoning ceiling with same architectural weaknesses | Benchmark only; not preferred |
|
||||
| NuExtract 1.5 3.8B | Literal schema filling | Limited implicit market reasoning; adds another generative runtime | Optional benchmark/fallback |
|
||||
| NuExtract 1.5 Smol 1.7B | Compact long-form extraction | Still autoregressive and not a sentiment/impact model | Optional CPU/on-demand filing stage |
|
||||
| NuExtract Tiny 0.5B | Very small extraction experiments | Accuracy ceiling too low for authoritative trading inputs without task tuning | Research/fine-tuning baseline |
|
||||
| GLiNER2 Large 340M | CPU-first entities, classes, relations, spans | Needs calibration and task-specific tuning for best results | Primary fast-path specialist |
|
||||
| FinBERT | Financial positive/negative/neutral probabilities | Not an extractor or reasoner | Per-company evidence sentiment |
|
||||
| Hybrid specialist + 9B | Routine precision plus retained hard-case intelligence | More engineering and observability work | Recommended architecture |
|
||||
| Hybrid + historical impact model | Text intelligence plus actual market-response learning | Requires leakage-safe dataset and monitoring | Best end-state |
|
||||
|
||||
## Highest-Priority Existing Problems
|
||||
|
||||
1. `services/extractor/vllm_client.py` ignores the supplied schema and requests only a generic JSON object.
|
||||
2. The vLLM default extraction temperature is `0.7`.
|
||||
3. Unknown provider values silently route to Ollama.
|
||||
4. Documents are truncated to 8,000 characters.
|
||||
5. The model is asked to invent authoritative novelty, confidence, impact, and horizon values.
|
||||
6. Those self-scores directly affect aggregation weighting.
|
||||
7. Provider attribution is hardcoded to Ollama.
|
||||
8. The extractor processes one job at a time at the application layer.
|
||||
9. Endpoint/model defaults conflict across code, migrations, Helm, and the vLLM deployment.
|
||||
10. A tracked Helm override contains plaintext production-like credentials and requires immediate rotation.
|
||||
|
||||
## Expected Performance Shape
|
||||
|
||||
The following are design targets to validate, not promises:
|
||||
|
||||
- 60-80% of representative documents accepted through the CPU fast path after calibration.
|
||||
- 2x or greater reduction in GPU-seconds per accepted document.
|
||||
- Peak GPU memory near the current 9B deployment because no second generative model is permanently GPU-resident.
|
||||
- p50 latency substantially lower for routine documents.
|
||||
- p95 latency near the current model path for adjudicated documents.
|
||||
- Better exact-field and evidence accuracy from deterministic/specialist stages.
|
||||
- Same broad semantic ceiling because the 9B model remains available.
|
||||
- Better impact/horizon calibration once the historical outcome model is approved.
|
||||
|
||||
## Immediate Next Decision
|
||||
|
||||
The first implementation milestone should not be GLiNER integration. It should be:
|
||||
|
||||
1. Rotate exposed credentials.
|
||||
2. Establish the real runtime model/configuration.
|
||||
3. Fix strict JSON Schema output and temperature on the current 9B endpoint.
|
||||
4. Build the gold corpus and replay harness.
|
||||
5. Then implement the gateway and specialist shadow path.
|
||||
|
||||
That order creates a fair baseline and prevents the project from attributing simple request fixes to the new architecture.
|
||||
@@ -0,0 +1,792 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
Intelligence Pipeline v3 replaces a monolithic "article to final trading-oriented JSON" request with a staged evidence and prediction architecture. The existing 9B model remains available, but its role changes from universal extractor and self-scorer to **semantic adjudicator** for the minority of documents that need broad language understanding.
|
||||
|
||||
The design intentionally chooses the best long-term architecture rather than the minimum code change:
|
||||
|
||||
- Generic OpenAI-compatible support is implemented as a capability-aware gateway, not another provider branch.
|
||||
- Explicit facts, entities, numbers, and sentiment are produced by CPU-first specialist components.
|
||||
- Novelty comes from retrieval and similarity.
|
||||
- Confidence comes from empirical calibration.
|
||||
- Impact and horizon come from a stock-specific model trained against realized outcomes.
|
||||
- The existing 9B vLLM model handles ambiguity, causality, implication, and conflicts.
|
||||
- Every field retains source evidence and model lineage.
|
||||
|
||||
## Repository Review Findings
|
||||
|
||||
The following findings materially shaped this design:
|
||||
|
||||
| Finding | Repository location | Consequence |
|
||||
|---|---|---|
|
||||
| The vLLM client receives a JSON Schema but sends only `response_format: {"type": "json_object"}`. | `services/extractor/vllm_client.py:63-91` | The server is not constraining generation to the actual schema. |
|
||||
| vLLM extraction defaults to temperature `0.7`. | `services/shared/config.py:64-71`, `services/shared/config.py:284-291` | Routine extraction is needlessly stochastic. |
|
||||
| Unknown provider values silently fall back to Ollama. | `services/extractor/llm_factory.py:1-6`, `services/extractor/llm_factory.py:47-67` | Configuration mistakes can invoke the wrong endpoint without failing. |
|
||||
| Long documents are truncated to the first 8,000 characters. | `services/extractor/prompts.py:102-105` | Filings, transcripts, and long articles can lose material facts. |
|
||||
| The prompt asks one model for summary, entities, relevance, sentiment, impact, horizon, novelty, confidence, and evidence. | `services/extractor/prompts.py:107-126` | Extraction, reasoning, prediction, and self-evaluation are coupled. |
|
||||
| The prompt supplies tracked tickers and invites inferred sector/theme exposure. | `services/extractor/prompts.py:85-98` | Explicit mentions and inferred exposure are mixed before evidence validation. |
|
||||
| Persisted provider attribution is hardcoded to `ollama`, including failures. | `services/extractor/worker.py:166-184`, `services/extractor/worker.py:227-244` | Audit and model-performance attribution are incorrect for vLLM. |
|
||||
| A single worker loop pops and processes one job at a time. | `services/extractor/main.py:438-468`, invocation near `services/extractor/main.py:628` | Application-level parallelism is constrained even if vLLM supports batching. |
|
||||
| Runtime refresh mutates a client's private `_config`. | `services/extractor/main.py:496-531` | The protocol does not expose lifecycle or reconfiguration cleanly. |
|
||||
| The thesis rewriter reimplements Ollama/vLLM branching. | `services/recommendation/thesis_llm.py:87-200` | Provider support is duplicated and will continue drifting. |
|
||||
| Model defaults conflict across Python config, database migrations, Helm values, and the standalone vLLM deployment. | `services/shared/config.py`, `infra/migrations`, `infra/helm/stonks-oracle/values.yaml`, `infra/kube-vllm/deployment.yaml` | The repository cannot prove which model is canonical at runtime. |
|
||||
| Model-produced novelty and confidence directly affect aggregation weight; model-produced impact is reused as sentiment strength and impact. | `services/aggregation/scoring.py:436-529`, `services/aggregation/worker.py:430-463` | Uncalibrated model self-scores can materially influence downstream signals. |
|
||||
| A tracked Helm override contains plaintext production-like credentials. | `infra/helm/stonks-oracle/values-live-math.yaml` | Immediate rotation and history remediation are required before feature work ships. |
|
||||
|
||||
The existing test suite around the LLM clients is useful. The focused provider tests passed after installing the declared dependencies plus the missing property-test dependency, but they encode current behavior and do not test true schema-constrained vLLM output.
|
||||
|
||||
## Decision Summary
|
||||
|
||||
### 1. Add generic OpenAI-compatible support
|
||||
|
||||
Yes, but do not add an `OpenAIClient` beside `VLLMClient` and `OllamaClient`. Rename the concept:
|
||||
|
||||
- `OllamaNativeClient` for `/api/chat` and Ollama-specific controls.
|
||||
- `OpenAICompatibleClient` for `/v1/chat/completions` and optionally `/v1/responses` after a separate compatibility gate.
|
||||
- `SpecialistHttpClient` for typed non-generative endpoints.
|
||||
|
||||
`vllm` becomes a profile alias whose protocol is `openai_chat`. Hosted OpenAI, LM Studio, SGLang, LocalAI, or another compatible server can be represented by endpoint capabilities rather than new `if provider == ...` branches.
|
||||
|
||||
Use direct `httpx` requests in the generic layer. This keeps the wire payload explicit, permits provider-specific `extra_body`, simplifies redacted request auditing, and avoids binding every compatible server to one SDK's assumptions.
|
||||
|
||||
### 2. Retain the 9B model, but stop using it for every stage
|
||||
|
||||
The RTX 4070 Ti SUPER remains dedicated to one 9B-class vLLM deployment. This preserves the current semantic ceiling and current peak VRAM class. The 9B model is invoked only for ambiguous cases and receives a compressed evidence packet rather than the raw entire document and ticker universe.
|
||||
|
||||
### 3. Use a CPU-first fast path
|
||||
|
||||
Run the following on CPU nodes:
|
||||
|
||||
- GLiNER2 Large candidate for entities, event classes, relations, and schema-oriented extraction.
|
||||
- FinBERT candidate for company-specific positive/negative/neutral probabilities.
|
||||
- Deterministic parsers and the existing symbol registry for numeric facts and ticker identity.
|
||||
- Compact embeddings plus fingerprints for novelty and deduplication.
|
||||
- A calibrated tabular impact model for market direction, magnitude, and horizon.
|
||||
|
||||
NuExtract 1.5 Smol is retained as an evaluated optional stage for long-form or hierarchical extraction. It is not made a second always-resident GPU model because the intended deployment should preserve the 9B model's GPU footprint and because GLiNER2 plus deterministic parsing may already cover most literal extraction.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Normalized document] --> B[Segmenter and offset map]
|
||||
B --> C[Deterministic candidates\ncompany aliases, tickers, numbers, dates]
|
||||
B --> D[GLiNER2 specialist\nentities, events, relations, facts]
|
||||
C --> E[Symbol resolver]
|
||||
D --> E
|
||||
E --> F[Evidence linker and verifier]
|
||||
F --> G[FinBERT per-company sentiment]
|
||||
F --> H[Novelty and dedup retrieval]
|
||||
G --> I[Confidence calibrator]
|
||||
H --> I
|
||||
I --> J{Fast-path acceptance?}
|
||||
J -->|yes| K[Approved evidence graph]
|
||||
J -->|no| L[9B Qwen adjudicator on vLLM]
|
||||
L --> M[Post-adjudication verifier]
|
||||
M --> K
|
||||
K --> N[Stock-specific impact model]
|
||||
N --> O[v3 intelligence records]
|
||||
O --> P[Compatibility adapter]
|
||||
P --> Q[Current aggregation and recommendation consumers]
|
||||
```
|
||||
|
||||
### Why this can be more intelligent without a larger footprint
|
||||
|
||||
A monolithic 9B model is broadly intelligent but is not necessarily the best estimator for every subproblem. The v3 design keeps that model for tasks requiring broad semantics while giving narrower jobs to components whose output can be calibrated and verified. The impact model adds information the language model does not have: observed historical market response. The result is not merely a smaller extractor; it is a system that combines textual reasoning with market-specific learned behavior.
|
||||
|
||||
## Component Design
|
||||
|
||||
### A. Inference Gateway
|
||||
|
||||
#### Package layout
|
||||
|
||||
```text
|
||||
services/shared/inference/
|
||||
├── protocol.py
|
||||
├── models.py
|
||||
├── registry.py
|
||||
├── router.py
|
||||
├── capabilities.py
|
||||
├── errors.py
|
||||
├── redaction.py
|
||||
└── clients/
|
||||
├── ollama_native.py
|
||||
├── openai_compatible.py
|
||||
└── specialist_http.py
|
||||
```
|
||||
|
||||
#### Core types
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ProviderCapabilities:
|
||||
chat_completions: bool
|
||||
responses_api: bool
|
||||
json_schema: bool
|
||||
json_object: bool
|
||||
seed: bool
|
||||
usage: bool
|
||||
max_completion_tokens: bool
|
||||
reasoning_toggle: bool
|
||||
model_listing: bool
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InferenceTarget:
|
||||
endpoint_id: UUID
|
||||
deployment_id: UUID
|
||||
protocol: Literal["ollama_native", "openai_chat", "specialist_http"]
|
||||
base_url: str
|
||||
model: str
|
||||
capabilities: ProviderCapabilities
|
||||
auth_secret_ref: str | None
|
||||
extra_headers: Mapping[str, str]
|
||||
extra_body: Mapping[str, Any]
|
||||
|
||||
@dataclass
|
||||
class StructuredGenerationRequest:
|
||||
messages: list[ChatMessage]
|
||||
json_schema: dict[str, Any] | None
|
||||
max_output_tokens: int
|
||||
temperature: float = 0.0
|
||||
seed: int | None = 0
|
||||
timeout_seconds: float = 120.0
|
||||
trace_id: str = ""
|
||||
|
||||
@dataclass
|
||||
class InferenceResult:
|
||||
content: str
|
||||
parsed: dict[str, Any] | None
|
||||
target: InferenceTarget
|
||||
structured_mode: Literal["json_schema", "json_object", "prompt_only", "none"]
|
||||
latency_ms: int
|
||||
input_tokens: int | None
|
||||
output_tokens: int | None
|
||||
request_id: str | None
|
||||
repaired: bool
|
||||
retries: int
|
||||
```
|
||||
|
||||
#### OpenAI-compatible structured output
|
||||
|
||||
The client chooses the strongest declared mode:
|
||||
|
||||
1. `json_schema`: send the actual schema and strict mode.
|
||||
2. `json_object`: allow only if the deployment profile explicitly permits it.
|
||||
3. `prompt_only`: allow only for experiments or legacy fallback.
|
||||
|
||||
For current vLLM versions, the gateway should support both standard `response_format` JSON Schema and a configurable vLLM `structured_outputs` extra body because deployed versions may differ. The endpoint profile records which wire form passed its capability probe.
|
||||
|
||||
Example standard payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "AxionML/Qwen3.5-9B-NVFP4",
|
||||
"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}],
|
||||
"temperature": 0,
|
||||
"max_tokens": 1536,
|
||||
"response_format": {
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "adjudication_response",
|
||||
"strict": true,
|
||||
"schema": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The gateway validates the parsed response again locally. Wire constraints reduce malformed output; they do not replace semantic validation.
|
||||
|
||||
### B. Endpoint Registry
|
||||
|
||||
#### New tables
|
||||
|
||||
```sql
|
||||
CREATE TABLE inference_endpoints (
|
||||
id UUID PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
protocol TEXT NOT NULL CHECK (protocol IN ('ollama_native','openai_chat','specialist_http')),
|
||||
base_url TEXT NOT NULL,
|
||||
auth_secret_ref TEXT,
|
||||
auth_scheme TEXT NOT NULL DEFAULT 'bearer',
|
||||
default_headers JSONB NOT NULL DEFAULT '{}',
|
||||
health_path TEXT,
|
||||
enabled BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
revision INTEGER NOT NULL DEFAULT 1,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE model_deployments (
|
||||
id UUID PRIMARY KEY,
|
||||
endpoint_id UUID NOT NULL REFERENCES inference_endpoints(id),
|
||||
served_model_name TEXT NOT NULL,
|
||||
display_name TEXT NOT NULL,
|
||||
capabilities JSONB NOT NULL,
|
||||
context_window INTEGER,
|
||||
max_output_tokens INTEGER,
|
||||
quantization TEXT,
|
||||
runtime_metadata JSONB NOT NULL DEFAULT '{}',
|
||||
enabled BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
revision INTEGER NOT NULL DEFAULT 1,
|
||||
UNIQUE(endpoint_id, served_model_name)
|
||||
);
|
||||
|
||||
CREATE TABLE agent_stage_bindings (
|
||||
id UUID PRIMARY KEY,
|
||||
agent_id UUID NOT NULL REFERENCES ai_agents(id),
|
||||
stage TEXT NOT NULL,
|
||||
model_deployment_id UUID REFERENCES model_deployments(id),
|
||||
route_order INTEGER NOT NULL DEFAULT 0,
|
||||
routing_config JSONB NOT NULL DEFAULT '{}',
|
||||
is_active BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
revision INTEGER NOT NULL DEFAULT 1,
|
||||
UNIQUE(agent_id, stage, route_order)
|
||||
);
|
||||
```
|
||||
|
||||
Authentication values are not stored in these tables. `auth_secret_ref` identifies a mounted secret or environment key understood by the deployment.
|
||||
|
||||
### C. Document Segmenter
|
||||
|
||||
The segmenter replaces the 8,000-character prefix truncation.
|
||||
|
||||
#### Output
|
||||
|
||||
```python
|
||||
class DocumentChunk(BaseModel):
|
||||
chunk_id: str
|
||||
document_id: UUID
|
||||
document_type: str
|
||||
section_path: list[str]
|
||||
speaker: str | None
|
||||
start_char: int
|
||||
end_char: int
|
||||
text: str
|
||||
overlap_left: int
|
||||
overlap_right: int
|
||||
boilerplate_score: float
|
||||
```
|
||||
|
||||
Suggested initial limits:
|
||||
|
||||
| Document type | Target chunk size | Overlap | Notes |
|
||||
|---|---:|---:|---|
|
||||
| News / press release | 700-1,000 tokens | 100 tokens | Preserve paragraph boundaries. |
|
||||
| Filing | 900-1,300 tokens | 150 tokens | Preserve headings and item sections. |
|
||||
| Transcript | 700-1,000 tokens | 100 tokens | Preserve speaker turns. |
|
||||
| Macro event | 500-800 tokens | 80 tokens | Favor compact event context. |
|
||||
|
||||
The final values are benchmark parameters, not hard-coded assumptions.
|
||||
|
||||
### D. Candidate Extraction and Symbol Resolution
|
||||
|
||||
Deterministic parsers generate high-precision candidates before specialist inference:
|
||||
|
||||
- Ticker tokens and exchange-qualified symbols.
|
||||
- Currency and number expressions, including `million`, `billion`, ranges, percentages, basis points, and per-share amounts.
|
||||
- Calendar and fiscal periods.
|
||||
- Comparison cues such as `up`, `down`, `beat`, `miss`, `raised`, `cut`, `above`, and `below`.
|
||||
- Company aliases from the symbol registry.
|
||||
|
||||
GLiNER2 receives focused schemas and returns spans for entities, event classes, relations, and structured facts. The resolver merges deterministic and specialist candidates using source offsets, aliases, and local context.
|
||||
|
||||
Explicit mentions and inferred exposures are different edge types:
|
||||
|
||||
```text
|
||||
Document --explicitly_mentions--> Company
|
||||
Event --directly_affects--> Company
|
||||
Event --inferred_exposure--> Company
|
||||
Company --competes_with--> Company
|
||||
Company --supplies--> Company
|
||||
```
|
||||
|
||||
Only explicit and verified direct effects enter the primary company extraction. Inferred exposure continues through the existing interpolation/propagation architecture with separate confidence and provenance.
|
||||
|
||||
### E. Sentiment Stage
|
||||
|
||||
FinBERT is run on company-linked evidence groups rather than the entire article. Each record contains:
|
||||
|
||||
```python
|
||||
class CompanySentiment(BaseModel):
|
||||
company_id: UUID
|
||||
evidence_ids: list[UUID]
|
||||
positive_probability: float
|
||||
negative_probability: float
|
||||
neutral_probability: float
|
||||
calibration_version: str
|
||||
model_deployment_id: UUID
|
||||
```
|
||||
|
||||
Mixed sentiment is computed from multiple evidence groups and disagreement. It is not an unconstrained fourth softmax label.
|
||||
|
||||
### F. Novelty Stage
|
||||
|
||||
Novelty becomes retrieval-based:
|
||||
|
||||
1. Compute an exact/near-duplicate fingerprint of normalized content.
|
||||
2. Embed document chunks and canonical company-event representations.
|
||||
3. Search a recent window in the vector index.
|
||||
4. Calculate document novelty and event novelty from nearest-neighbor similarity, duplicate count, source timing, and event identity.
|
||||
5. Store nearest matches for explainability.
|
||||
|
||||
A compact embedding model will be selected in the evaluation harness. The implementation must keep the embedding backend replaceable and must not entangle novelty scoring with the generative endpoint.
|
||||
|
||||
### G. Confidence and Routing
|
||||
|
||||
#### Confidence features
|
||||
|
||||
- Entity span score.
|
||||
- Alias-resolution margin between first and second candidate.
|
||||
- Numeric parser validity.
|
||||
- Evidence coverage.
|
||||
- Relation score.
|
||||
- Sentiment calibration confidence.
|
||||
- Cross-stage agreement.
|
||||
- Duplicate/novelty certainty.
|
||||
- Document completeness.
|
||||
- Document type and known hard-case patterns.
|
||||
|
||||
A calibration artifact maps these features to field-level correctness probabilities. Routing uses both calibrated confidence and hard rules.
|
||||
|
||||
#### Example adjudication triggers
|
||||
|
||||
```text
|
||||
UNRESOLVED_ALIAS
|
||||
MULTIPLE_PRIMARY_COMPANIES
|
||||
CONTRADICTORY_NUMERIC_FACTS
|
||||
CONFLICTING_SENTIMENT
|
||||
IMPLIED_CAUSAL_IMPACT
|
||||
GUIDANCE_VS_CONSENSUS_REQUIRES_REASONING
|
||||
MATERIAL_FIELD_MISSING
|
||||
EVIDENCE_COVERAGE_BELOW_THRESHOLD
|
||||
CALIBRATED_CONFIDENCE_BELOW_THRESHOLD
|
||||
LONG_DOCUMENT_CROSS_CHUNK_RELATION
|
||||
```
|
||||
|
||||
The desired initial target is 60-80 percent Fast_Path coverage after calibration. This is an evaluation target, not an assumed result.
|
||||
|
||||
### H. Adjudication Packet
|
||||
|
||||
The 9B model receives only the information necessary to resolve a specific ambiguity:
|
||||
|
||||
```python
|
||||
class AdjudicationPacket(BaseModel):
|
||||
document_id: UUID
|
||||
document_type: str
|
||||
question_codes: list[str]
|
||||
candidate_companies: list[CompanyCandidate]
|
||||
candidate_events: list[EventCandidate]
|
||||
candidate_facts: list[FactCandidate]
|
||||
candidate_sentiments: list[CompanySentiment]
|
||||
evidence_spans: list[EvidenceSpan]
|
||||
relevant_chunks: list[DocumentChunk]
|
||||
required_decisions: list[str]
|
||||
```
|
||||
|
||||
The adjudicator output does not contain final novelty, confidence, impact, or horizon. It resolves candidate identity, relationship, event interpretation, and supported qualitative direction. All output is evidence-linked and revalidated.
|
||||
|
||||
### I. Impact and Horizon Model
|
||||
|
||||
This is the highest-impact stock-specific change.
|
||||
|
||||
#### Problem decomposition
|
||||
|
||||
- **Text sentiment**: What tone or directional implication is supported by the document?
|
||||
- **Event identity**: What happened?
|
||||
- **Market impact**: How has this kind of event historically affected this kind of security in this market regime?
|
||||
- **Horizon**: Over what time window did the response usually manifest or decay?
|
||||
|
||||
The current model conflates these. v3 separates them.
|
||||
|
||||
#### Features
|
||||
|
||||
- Event class probability vector.
|
||||
- Company-specific sentiment probability vector.
|
||||
- Numeric magnitude and normalized surprise when consensus or prior value exists.
|
||||
- Source credibility and historical source accuracy.
|
||||
- Novelty and duplicate count.
|
||||
- Evidence coverage and extraction uncertainty.
|
||||
- Document type.
|
||||
- Company sector, industry, market-cap bucket, liquidity, and beta.
|
||||
- Pre-event volatility, volume regime, and broad market regime.
|
||||
- Whether the event is direct, second-order, confirmed, quoted, or speculative.
|
||||
|
||||
#### Labels
|
||||
|
||||
Generate leakage-safe targets at defined event timestamps:
|
||||
|
||||
- Signed abnormal return relative to an approved benchmark.
|
||||
- Absolute abnormal move.
|
||||
- Abnormal volume.
|
||||
- Direction labels for intraday, 1d, 7d, 30d, and 90d windows.
|
||||
- Time to peak response and decay where data quality supports it.
|
||||
|
||||
#### Model family
|
||||
|
||||
Start with a deterministic event-weight baseline plus a CPU tabular learner such as gradient-boosted trees. Calibrate class probabilities out-of-time. The model artifact and feature pipeline are versioned independently.
|
||||
|
||||
The compatibility adapter may initially map expected signed magnitude to current `impact_score` and the most probable horizon to current `impact_horizon`, but richer distributions remain available to new consumers.
|
||||
|
||||
### J. V3 Storage Schema
|
||||
|
||||
Suggested logical records:
|
||||
|
||||
```python
|
||||
class EvidenceSpan(BaseModel):
|
||||
id: UUID
|
||||
document_id: UUID
|
||||
chunk_id: str
|
||||
start_char: int
|
||||
end_char: int
|
||||
text: str
|
||||
checksum: str
|
||||
|
||||
class ExtractedEntity(BaseModel):
|
||||
id: UUID
|
||||
entity_type: str
|
||||
literal_text: str
|
||||
canonical_id: UUID | None
|
||||
evidence_id: UUID
|
||||
confidence: float
|
||||
derivation: str
|
||||
|
||||
class ExtractedFact(BaseModel):
|
||||
id: UUID
|
||||
fact_type: str
|
||||
subject_entity_id: UUID | None
|
||||
predicate: str
|
||||
literal_value: str
|
||||
normalized_value: dict | None
|
||||
period: dict | None
|
||||
evidence_ids: list[UUID]
|
||||
confidence: float
|
||||
derivation: str
|
||||
|
||||
class CompanySignalCandidate(BaseModel):
|
||||
company_id: UUID
|
||||
relevance_probability: float
|
||||
event_probabilities: dict[str, float]
|
||||
sentiment_probabilities: dict[str, float]
|
||||
direction_probabilities: dict[str, float]
|
||||
horizon_probabilities: dict[str, float]
|
||||
expected_magnitude: float | None
|
||||
evidence_ids: list[UUID]
|
||||
routing_reasons: list[str]
|
||||
adjudicated: bool
|
||||
|
||||
class StageLineage(BaseModel):
|
||||
stage: str
|
||||
endpoint_id: UUID | None
|
||||
deployment_id: UUID | None
|
||||
model_version: str | None
|
||||
schema_version: str
|
||||
calibration_version: str | None
|
||||
started_at: datetime
|
||||
duration_ms: int
|
||||
status: str
|
||||
```
|
||||
|
||||
### K. Compatibility Adapter
|
||||
|
||||
The adapter creates current records without discarding v3 provenance:
|
||||
|
||||
| Current field | V3 source |
|
||||
|---|---|
|
||||
| `summary` | Deterministic template or optional 9B narrative generated from approved facts. |
|
||||
| `macro_themes` | Approved event/theme classes. |
|
||||
| `novelty_score` | Retrieval-derived event/document novelty. |
|
||||
| `confidence` | Calibrated record correctness probability. |
|
||||
| `ticker` | Canonical symbol registry resolution. |
|
||||
| `relevance` | Calibrated direct-relevance probability. |
|
||||
| `sentiment` | Company-specific calibrated distribution mapped to legacy enum. |
|
||||
| `impact_score` | Approved impact-model magnitude mapped to legacy range. |
|
||||
| `impact_horizon` | Most probable approved horizon. |
|
||||
| `catalyst_type` | Versioned event taxonomy mapping. |
|
||||
| `evidence_spans` | Exact source spans. |
|
||||
|
||||
The adapter marks `model_provider = 'hybrid'` and stores complete stage lineage separately. No provider identity is hardcoded.
|
||||
|
||||
## Deployment Design for RTX 4070 Ti SUPER Cluster
|
||||
|
||||
### GPU deployment
|
||||
|
||||
One vLLM pod remains on the RTX 4070 Ti SUPER:
|
||||
|
||||
```yaml
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
nodeSelector:
|
||||
accelerator: rtx-4070-ti-super
|
||||
args:
|
||||
- --model
|
||||
- AxionML/Qwen3.5-9B-NVFP4
|
||||
- --served-model-name
|
||||
- stonks-adjudicator-9b
|
||||
- --max-model-len
|
||||
- "8192"
|
||||
- --max-num-seqs
|
||||
- "8"
|
||||
- --gpu-memory-utilization
|
||||
- "0.80"
|
||||
- --structured-outputs-config.backend
|
||||
- auto
|
||||
```
|
||||
|
||||
Exact flags must match the pinned vLLM version. The deployment test must verify strict schema output before promotion.
|
||||
|
||||
### CPU specialist deployment
|
||||
|
||||
```yaml
|
||||
replicas: 2
|
||||
resources:
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
limits:
|
||||
cpu: "6"
|
||||
memory: 10Gi
|
||||
```
|
||||
|
||||
Initial pod contents:
|
||||
|
||||
- GLiNER2 Large.
|
||||
- FinBERT.
|
||||
- Tokenizers and deterministic parsers.
|
||||
- Optional embedding model.
|
||||
|
||||
NuExtract 1.5 Smol should run as a separate benchmark or on-demand CPU deployment so its value can be measured independently.
|
||||
|
||||
### Queue topology
|
||||
|
||||
```text
|
||||
extraction.incoming
|
||||
-> intelligence.router
|
||||
-> extraction.fast
|
||||
-> extraction.adjudication
|
||||
-> extraction.persist
|
||||
-> extraction.review
|
||||
```
|
||||
|
||||
The router owns document state transitions. Workers use leases and idempotency keys so a retry cannot create duplicate intelligence records.
|
||||
|
||||
### Concurrency
|
||||
|
||||
- Fast path: configurable worker pool, initially 4-8 concurrent documents per pod.
|
||||
- Specialist API: micro-batching bounded by maximum wait time.
|
||||
- Adjudicator: application semaphore aligned with vLLM `max-num-seqs` and measured KV-cache behavior.
|
||||
- Persistence: independent bounded pool.
|
||||
|
||||
## OpenAI-Compatible Support Details
|
||||
|
||||
### Profiles
|
||||
|
||||
| Profile | Protocol | Typical use |
|
||||
|---|---|---|
|
||||
| `ollama` | `ollama_native` | Existing Ollama endpoint. |
|
||||
| `vllm` | `openai_chat` | Backward-compatible alias using vLLM capability profile. |
|
||||
| `openai` | `openai_chat` | Hosted OpenAI endpoint with secret reference and egress policy. |
|
||||
| `openai_compatible` | `openai_chat` | Any explicitly configured compatible server. |
|
||||
| `specialist` | `specialist_http` | Typed GLiNER/FinBERT service. |
|
||||
|
||||
### Capability probes
|
||||
|
||||
A deployment activation test performs:
|
||||
|
||||
1. Health request.
|
||||
2. Model listing if supported.
|
||||
3. Minimal chat request.
|
||||
4. Strict JSON Schema request.
|
||||
5. Usage metadata check.
|
||||
6. Seed/determinism check if declared.
|
||||
7. Maximum output field compatibility check.
|
||||
|
||||
Probe results are stored with timestamp and software version. A failed capability cannot be enabled merely by selecting it in the UI.
|
||||
|
||||
### Egress and data policy
|
||||
|
||||
Endpoint profiles include data-handling classification:
|
||||
|
||||
```text
|
||||
local_private
|
||||
cluster_private
|
||||
approved_external
|
||||
forbidden_for_sensitive_docs
|
||||
```
|
||||
|
||||
External endpoints are disabled by default. Routing to an approved external endpoint requires both an active binding and a document policy permitting egress.
|
||||
|
||||
## Evaluation Strategy
|
||||
|
||||
### Gold corpus
|
||||
|
||||
Build a minimum initial corpus of 1,000 human-reviewed documents, stratified across:
|
||||
|
||||
- News, filings, transcripts, and press releases.
|
||||
- Single-company and multi-company stories.
|
||||
- Earnings beats/misses and guidance changes.
|
||||
- M&A, legal, regulatory, product, supply-chain, rating, management, and macro events.
|
||||
- Explicit facts versus implied consequences.
|
||||
- Short and long documents.
|
||||
- Duplicate and recycled stories.
|
||||
|
||||
### Compared systems
|
||||
|
||||
1. Current production path with current model and current prompt.
|
||||
2. Current 9B model with corrected temperature and strict JSON Schema.
|
||||
3. GLiNER2 + deterministic extraction.
|
||||
4. GLiNER2 + deterministic extraction + FinBERT.
|
||||
5. Optional NuExtract 1.5 Smol extraction path.
|
||||
6. Full v3 fast path.
|
||||
7. Full v3 with 9B adjudication.
|
||||
|
||||
This separation prevents architecture gains from being confused with a simple fix to the current vLLM request.
|
||||
|
||||
### Metrics
|
||||
|
||||
- Company/ticker precision, recall, F1.
|
||||
- Event macro-F1 and per-class F1.
|
||||
- Numeric exact match with tolerance-aware normalization.
|
||||
- Relation F1.
|
||||
- Evidence support and offset validity.
|
||||
- Company-specific sentiment macro-F1.
|
||||
- Confidence ECE and Brier score.
|
||||
- Unsupported-claim rate.
|
||||
- JSON/schema failure rate.
|
||||
- Fast-path coverage.
|
||||
- Adjudication reason distribution.
|
||||
- p50/p95 latency and documents per minute.
|
||||
- CPU-seconds, GPU-seconds, tokens, and peak GPU memory per document.
|
||||
- Impact-model direction accuracy, calibration, rank correlation, and out-of-time error by horizon.
|
||||
|
||||
### Promotion sequence
|
||||
|
||||
1. Correct current vLLM schema constraints and temperature; establish baseline.
|
||||
2. Run v3 offline replay.
|
||||
3. Run v3 shadow mode.
|
||||
4. Enable v3 for audit-only UI.
|
||||
5. Canary compatibility outputs for a small percentage of non-trading downstream traffic.
|
||||
6. Canary signal influence with automatic rollback.
|
||||
7. Promote by document type and confidence tier.
|
||||
8. Retire v2 only after a separately reviewed milestone.
|
||||
|
||||
## Security Design
|
||||
|
||||
### Immediate blocker
|
||||
|
||||
The repository contains plaintext production-like credentials in a tracked Helm values file. Treat them as compromised:
|
||||
|
||||
1. Rotate the database, object-store, Redis, broker, and market-data credentials.
|
||||
2. Disable or replace old keys.
|
||||
3. Remove secret values from tracked files.
|
||||
4. Purge historical values using an approved Git history rewrite process.
|
||||
5. Migrate to an external secret manager.
|
||||
6. Add secret scanning to local hooks and CI.
|
||||
7. Audit access logs for the affected credentials.
|
||||
|
||||
No values are reproduced in this specification.
|
||||
|
||||
### Inference security
|
||||
|
||||
- Secrets are resolved only at runtime.
|
||||
- Request logging records hashes and metadata, not authorization values.
|
||||
- Raw source text is not logged at INFO level.
|
||||
- External endpoint use is policy-gated.
|
||||
- Stored raw prompts and responses use restricted object-store buckets and retention policies.
|
||||
- Provider errors are normalized to avoid echoing secret-bearing response headers.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit tests
|
||||
|
||||
- Endpoint capability selection.
|
||||
- Strict schema payload construction.
|
||||
- Header and error redaction.
|
||||
- Segment offset round trips.
|
||||
- Numeric normalization.
|
||||
- Alias resolution and ambiguity margins.
|
||||
- Evidence linkage.
|
||||
- Compatibility mappings.
|
||||
- Confidence feature construction.
|
||||
- Routing rules.
|
||||
|
||||
### Property-based tests
|
||||
|
||||
- Every Evidence_Span round-trips to identical source text.
|
||||
- Normalization never changes the literal stored value.
|
||||
- Unknown providers always fail closed.
|
||||
- Credentials never appear in serialized errors or logs.
|
||||
- Compatibility mappings remain bounded in legacy field ranges.
|
||||
- Reprocessing the same document and model versions is idempotent.
|
||||
- Route decisions are deterministic for identical calibrated inputs.
|
||||
|
||||
### Contract tests
|
||||
|
||||
- Ollama native endpoint.
|
||||
- vLLM OpenAI-compatible endpoint.
|
||||
- Mock hosted OpenAI-compatible endpoint.
|
||||
- Specialist service schemas.
|
||||
- Capability probe behavior across supported structured-output modes.
|
||||
|
||||
### Integration tests
|
||||
|
||||
- Full document through fast path.
|
||||
- Full document through adjudication path.
|
||||
- Adjudicator outage with safe fast-path handling.
|
||||
- Long filing crossing multiple chunks.
|
||||
- Multi-company article with opposing sentiment.
|
||||
- Duplicate story and novelty calculation.
|
||||
- Rollback from v3 to v2.
|
||||
|
||||
### Load tests
|
||||
|
||||
- CPU specialist batching.
|
||||
- Queue backpressure.
|
||||
- vLLM concurrent adjudication.
|
||||
- Peak 4070 Ti SUPER memory.
|
||||
- End-to-end throughput at representative article arrival rates.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 0: Security and baseline
|
||||
|
||||
Rotate secrets, add scanning, pin current runtime configuration, and benchmark the existing path.
|
||||
|
||||
### Phase 1: Provider foundation
|
||||
|
||||
Add the Inference Gateway and registry. Convert current extractor, classifier, and thesis rewriter. Keep behavior otherwise equivalent.
|
||||
|
||||
### Phase 2: Correct current generative extraction
|
||||
|
||||
Use strict schema-constrained vLLM output, temperature zero, accurate provider lineage, and bounded output sizes. This produces a fair current baseline.
|
||||
|
||||
### Phase 3: V3 data and specialist shadow path
|
||||
|
||||
Add segmentation, v3 storage, deterministic extraction, GLiNER2, FinBERT, novelty, confidence, and audit UI. Persist shadow outputs only.
|
||||
|
||||
### Phase 4: Adjudication and impact model
|
||||
|
||||
Add confidence routing, focused 9B adjudication, historical outcome features, deterministic impact baseline, and trained impact model.
|
||||
|
||||
### Phase 5: Canary and promotion
|
||||
|
||||
Enable compatibility outputs by percentage and document type, then gradually allow v3 signals into aggregation.
|
||||
|
||||
### Phase 6: Fine-tuning and cleanup
|
||||
|
||||
Fine-tune specialist models from reviewed cases, increase fast-path coverage, then remove deprecated v2 code in a separate change.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Specialist model misses implicit meaning. | Retain 9B adjudication with calibrated routing. |
|
||||
| Complexity creates more failure modes. | Typed stage contracts, idempotent queues, stage-level metrics, and safe fallback. |
|
||||
| Fast-path confidence is overestimated. | Held-out calibration, conservative thresholds, and shadow review. |
|
||||
| Impact model learns leakage or regime artifacts. | Event-time feature snapshots, out-of-time validation, per-regime monitoring, and immutable predictions. |
|
||||
| OpenAI-compatible servers differ subtly. | Capability probes and profile-specific wire settings, not optimistic assumptions. |
|
||||
| A second model increases memory. | Keep specialists on CPU and make NuExtract optional/on-demand. |
|
||||
| Existing downstream code assumes one model output. | Compatibility adapter and additive migrations. |
|
||||
| Reviewer labels become inconsistent. | Annotation guide, double review for hard cases, and inter-annotator agreement tracking. |
|
||||
@@ -0,0 +1,314 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
Stonks Oracle currently asks a general-purpose generative model to perform entity discovery, ticker attribution, fact extraction, event classification, sentiment analysis, novelty estimation, confidence estimation, impact scoring, horizon selection, evidence quoting, and summarization in one response. That design is convenient, but it couples factual extraction to generative sampling and allows uncalibrated model self-assessments to influence signal weighting.
|
||||
|
||||
This specification introduces **Intelligence Pipeline v3**, a multi-stage, evidence-grounded inference system that preserves the current 9B model's reasoning ability for genuinely ambiguous documents while moving routine extraction, sentiment, novelty, confidence, and impact estimation into specialized and calibratable components. The target deployment retains the existing RTX 4070 Ti SUPER vLLM footprint and uses CPU-first specialist services for the fast path.
|
||||
|
||||
The specification also replaces provider-specific branching with a capability-aware inference gateway supporting Ollama native endpoints and generic OpenAI-compatible endpoints, including vLLM and hosted OpenAI-compatible services.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Match or exceed the current 9B pipeline's field-level accuracy and reasoning ceiling.
|
||||
2. Reduce average GPU work per document without increasing peak GPU memory materially.
|
||||
3. Make every extracted fact traceable to evidence in the source document.
|
||||
4. Replace model-generated confidence, novelty, impact, and horizon values with calibrated or deterministic values.
|
||||
5. Support generic OpenAI-compatible inference without adding another duplicated provider branch.
|
||||
6. Establish a measurable benchmark, shadow rollout, and promotion process.
|
||||
7. Preserve downstream compatibility while the v2 schema and database consumers are migrated.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
1. Replacing the existing recommendation, risk, or trading engines in one release.
|
||||
2. Removing the current 9B model before the v3 pipeline passes promotion gates.
|
||||
3. Treating backtest profit alone as proof of extraction correctness.
|
||||
4. Sending credentials or proprietary documents to external providers by default.
|
||||
5. Requiring a second GPU-resident generative model.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **Inference_Gateway**: Shared client and routing layer that invokes Ollama-native, OpenAI-compatible, and specialist inference endpoints through one typed interface.
|
||||
- **Endpoint_Profile**: Persisted endpoint configuration containing protocol, URL, authentication reference, capabilities, and health settings.
|
||||
- **Model_Deployment**: A model served by an Endpoint_Profile with declared capabilities and limits.
|
||||
- **Pipeline_Stage**: One step in Intelligence Pipeline v3, such as segmentation, extraction, sentiment, verification, novelty, adjudication, or impact prediction.
|
||||
- **Fast_Path**: CPU-first processing that completes without invoking the 9B generative model.
|
||||
- **Adjudication_Path**: Processing that invokes the 9B model because evidence is ambiguous, contradictory, incomplete, or semantically complex.
|
||||
- **Evidence_Span**: Exact source text plus stable character offsets and a chunk identifier.
|
||||
- **Candidate**: A proposed entity, fact, event, sentiment, or relation before validation and calibration.
|
||||
- **Calibrated_Confidence**: Probability-like confidence derived from validation data, not a number supplied by a generative model.
|
||||
- **Impact_Model**: A lightweight supervised model that estimates signed market impact and horizon from extracted features and historical outcomes.
|
||||
- **Compatibility_Adapter**: Mapper from v3 records to the current v2 `document_intelligence` and `document_impact_records` structures.
|
||||
- **Gold_Corpus**: Human-reviewed documents and field-level labels used for acceptance testing.
|
||||
- **Shadow_Mode**: Running v3 alongside the current pipeline without allowing v3 outputs to affect production decisions.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Secure Baseline and Credential Remediation
|
||||
|
||||
**User Story:** As an operator, I want repository and deployment credentials handled through secret stores, so that model-pipeline improvements do not ship on top of exposed credentials.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Team SHALL rotate every credential currently stored as plaintext in tracked repository files before deploying Intelligence Pipeline v3.
|
||||
2. THE Repository SHALL remove plaintext database, object-store, Redis, broker, and market-data credentials from tracked Helm values and Git history.
|
||||
3. THE Deployment SHALL reference credentials through Kubernetes Secrets populated by External Secrets, SOPS, Sealed Secrets, or an equivalent approved mechanism.
|
||||
4. THE CI_Pipeline SHALL run secret scanning on pull requests and protected branches.
|
||||
5. IF secret scanning detects a high-confidence credential, THEN THE CI_Pipeline SHALL fail before packaging or deployment.
|
||||
6. THE Documentation SHALL record the rotation date and affected secret names without recording secret values.
|
||||
|
||||
### Requirement 2: Capability-Aware Generic Inference Gateway
|
||||
|
||||
**User Story:** As a developer, I want one inference abstraction that supports Ollama and generic OpenAI-compatible services, so that endpoints can be changed without duplicating business logic.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Inference_Gateway SHALL support the protocols `ollama_native`, `openai_chat`, and `specialist_http`.
|
||||
2. THE Inference_Gateway SHALL treat `vllm` as a backward-compatible profile alias for `openai_chat`, not as a separate client implementation.
|
||||
3. WHEN an `openai_chat` request requires structured output and the endpoint declares `json_schema` support, THE Inference_Gateway SHALL send the complete supplied JSON Schema in strict structured-output mode.
|
||||
4. WHEN an endpoint supports only JSON-object mode, THE Inference_Gateway SHALL use JSON-object mode only when that fallback is explicitly enabled for the Model_Deployment.
|
||||
5. WHEN neither schema nor JSON-object constraints are supported, THE Inference_Gateway SHALL use prompt-only JSON generation only when explicitly enabled and SHALL mark the response as unconstrained.
|
||||
6. IF a provider or protocol value is unknown, THEN THE Inference_Gateway SHALL fail closed with a configuration error and SHALL NOT silently route to Ollama.
|
||||
7. THE Inference_Gateway SHALL support configurable base URL, request path, API-key secret reference, authorization scheme, additional headers, timeouts, retries, concurrency limit, and provider-specific extra request fields.
|
||||
8. THE Inference_Gateway SHALL redact authentication values and configured sensitive headers from logs, traces, and stored request snapshots.
|
||||
9. THE Inference_Gateway SHALL expose typed response metadata including endpoint ID, deployment ID, model name, protocol, request ID, latency, token usage, structured-output mode, retry count, and error category.
|
||||
10. THE Inference_Gateway SHALL provide health and capability probes and cache their results with a bounded TTL.
|
||||
11. WHEN endpoint capabilities are changed or a probe fails, THE Router SHALL invalidate the cached capability record before the next invocation.
|
||||
12. THE Existing thesis rewriter, event classifier, and document extractor SHALL use the same Inference_Gateway rather than implementing separate Ollama/vLLM branches.
|
||||
|
||||
### Requirement 3: Canonical Endpoint and Model Registry
|
||||
|
||||
**User Story:** As an operator, I want the database and UI to identify exactly which endpoint and model serve each stage, so that environment, migration, Helm, and runtime defaults cannot drift silently.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Database SHALL store `inference_endpoints`, `model_deployments`, and `agent_stage_bindings` as canonical runtime records.
|
||||
2. EACH Inference_Endpoint SHALL include name, protocol, base URL, authentication secret reference, health path, default headers, enabled state, and timestamps.
|
||||
3. EACH Model_Deployment SHALL include endpoint ID, served model name, display name, capabilities, context limit, output limit, quantization, structured-output modes, and enabled state.
|
||||
4. EACH Agent_Stage_Binding SHALL map an agent and pipeline stage to one or more ordered Model_Deployments plus routing configuration.
|
||||
5. WHEN runtime configuration is resolved, THE Service SHALL record the exact endpoint, deployment, and binding revision used.
|
||||
6. THE API SHALL validate endpoint URLs, protocol values, capability declarations, and model names before activation.
|
||||
7. THE UI SHALL use controlled protocol and endpoint selections rather than an unrestricted provider text field.
|
||||
8. THE Migration SHALL translate existing `ollama` and `vllm` agent settings into Endpoint_Profile and Model_Deployment records without breaking active agents.
|
||||
9. THE Application SHALL have one documented fallback configuration source; conflicting model defaults in code, migrations, and Helm SHALL be removed.
|
||||
|
||||
### Requirement 4: Document Segmentation and Source Preservation
|
||||
|
||||
**User Story:** As an analyst, I want long articles, filings, and transcripts processed without destructive truncation, so that material facts near the end of a document are not lost.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Pipeline SHALL preserve the full normalized source document and SHALL NOT truncate it to a fixed character prefix for extraction.
|
||||
2. THE Segmenter SHALL create sentence-aware chunks with stable chunk IDs, source character offsets, and configurable overlap.
|
||||
3. THE Segmenter SHALL use document-type-specific chunk sizes for articles, filings, transcripts, and press releases.
|
||||
4. THE Segmenter SHALL preserve headings, speaker labels, table-derived text markers, and section boundaries when present.
|
||||
5. WHEN duplicate or boilerplate sections are detected, THE Segmenter SHALL mark them without deleting the only occurrence of a fact.
|
||||
6. THE Pipeline SHALL retain a mapping from every downstream Evidence_Span to the original document offsets.
|
||||
7. IF a document cannot be decoded or segmented, THEN THE Pipeline SHALL mark the document as a typed preprocessing failure and SHALL NOT fabricate an empty extraction.
|
||||
|
||||
### Requirement 5: Deterministic Candidate Generation and Ticker Resolution
|
||||
|
||||
**User Story:** As a signal consumer, I want explicit companies and numeric facts resolved deterministically where possible, so that a language model is not asked to invent identifiers or parse trivial values.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Candidate_Generator SHALL detect explicit ticker symbols, company names, aliases, executives, products, currencies, percentages, dates, ranges, EPS values, revenue values, guidance values, and common financial ratios.
|
||||
2. THE Symbol_Resolver SHALL use the existing company and symbol registry as the source of truth for ticker identity.
|
||||
3. THE Pipeline SHALL distinguish explicit company mentions from inferred exposure relationships.
|
||||
4. THE Pipeline SHALL NOT pass the entire tracked-ticker universe to a generative prompt.
|
||||
5. WHEN multiple companies match an alias, THE Symbol_Resolver SHALL return ranked candidates and SHALL require contextual disambiguation or adjudication.
|
||||
6. WHEN a ticker is not present in the symbol registry, THE Pipeline SHALL preserve the literal mention as unresolved rather than inventing a registered ticker.
|
||||
7. THE Numeric_Normalizer SHALL retain both literal source text and normalized values, currencies, units, periods, and ranges.
|
||||
8. THE Pipeline SHALL reject normalized numeric facts whose value cannot be traced to an Evidence_Span.
|
||||
|
||||
### Requirement 6: Specialist Extraction Service
|
||||
|
||||
**User Story:** As an operator, I want routine entity, event, relation, and fact extraction to run on CPU-first specialist models, so that GPU capacity is reserved for difficult reasoning.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Specialist_Service SHALL expose batched APIs for entity extraction, schema extraction, relation extraction, and text classification.
|
||||
2. THE Initial specialist extractor SHALL support company, person, product, event, financial metric, date, percentage, currency, and relationship schemas.
|
||||
3. THE Specialist_Service SHALL return character spans and per-candidate scores for every extracted item.
|
||||
4. THE Specialist_Service SHALL run without requiring the RTX 4070 Ti SUPER.
|
||||
5. THE Initial deployment SHALL evaluate GLiNER2 Large as the primary unified extraction and classification model.
|
||||
6. THE Benchmark SHALL evaluate NuExtract 1.5 Smol as an optional long-form or hierarchical fact-extraction stage, but it SHALL NOT become an always-resident GPU model without passing incremental-value and resource gates.
|
||||
7. THE Specialist_Service SHALL support model version pinning, warm-up, health checks, bounded batching, and graceful degradation.
|
||||
8. WHEN specialist inference fails, THE Router SHALL either retry according to policy or route to adjudication; it SHALL record the failure and SHALL NOT silently substitute default facts.
|
||||
9. THE Specialist_Service SHALL expose model and schema versions in every response.
|
||||
|
||||
### Requirement 7: Company-Specific Financial Sentiment
|
||||
|
||||
**User Story:** As an analyst, I want sentiment tied to each company and supporting evidence, so that a positive statement about one firm is not applied to every company in the article.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Sentiment_Stage SHALL score evidence sentences or evidence groups associated with each resolved company.
|
||||
2. THE Initial sentiment classifier SHALL evaluate FinBERT as the baseline financial-domain model.
|
||||
3. THE Sentiment_Stage SHALL return positive, negative, and neutral probabilities rather than only a discrete label.
|
||||
4. THE Pipeline SHALL derive mixed sentiment from conflicting supported evidence, not from an unconstrained model label.
|
||||
5. WHEN an article mentions competitors with opposing effects, THE Pipeline SHALL produce separate company-specific sentiment records.
|
||||
6. THE Sentiment_Stage SHALL preserve the evidence IDs used for each probability distribution.
|
||||
7. THE Production model SHALL be calibrated on the Gold_Corpus before its probabilities are treated as confidence values.
|
||||
|
||||
### Requirement 8: Evidence Verification and Grounding
|
||||
|
||||
**User Story:** As an auditor, I want every material claim verified against source evidence, so that generated summaries and signals cannot rely on unsupported assertions.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. EVERY material company fact, event, amount, direction, and relationship SHALL reference one or more Evidence_Spans.
|
||||
2. THE Verifier SHALL check span validity, source offsets, entity association, and schema compatibility.
|
||||
3. THE Benchmark SHALL evaluate a compact entailment verifier for claims that require semantic validation beyond exact matching.
|
||||
4. IF a candidate conflicts with its evidence, THEN THE Pipeline SHALL reject it or route the conflict to adjudication.
|
||||
5. THE Pipeline SHALL calculate evidence coverage as the proportion of required fields supported by valid spans.
|
||||
6. THE Pipeline SHALL store rejected candidates and rejection reasons for audit and active learning.
|
||||
7. JSON repair SHALL NOT transform an unsupported or truncated generative answer into a valid production extraction without marking it as repaired and revalidating every material field.
|
||||
|
||||
### Requirement 9: Deterministic Novelty and Duplicate Detection
|
||||
|
||||
**User Story:** As a signal consumer, I want novelty based on comparison with recent information, so that a model's subjective novelty guess does not amplify repeated news.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Novelty_Stage SHALL compare each document and material event against a configurable recent-history window.
|
||||
2. THE Novelty_Stage SHALL combine exact/near-duplicate fingerprints with compact semantic embeddings.
|
||||
3. THE Pipeline SHALL calculate novelty separately for document-level content and company-event content.
|
||||
4. THE Novelty_Stage SHALL return nearest matching document or event IDs plus similarity scores.
|
||||
5. THE Pipeline SHALL derive `novelty_score` from the similarity distribution and duplicate count using a versioned deterministic formula or calibrated model.
|
||||
6. A generative model SHALL NOT provide the authoritative novelty value used by aggregation.
|
||||
7. WHEN novelty cannot be calculated because history is unavailable, THE Pipeline SHALL use a conservative versioned default and mark the reason.
|
||||
|
||||
### Requirement 10: Calibrated Extraction Confidence
|
||||
|
||||
**User Story:** As a downstream scorer, I want confidence to reflect observed correctness, so that the system does not trust a model merely because it reports confidence in itself.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Pipeline SHALL calculate field-level and record-level confidence from specialist scores, symbol resolution, evidence validation, schema completeness, model agreement, and historical calibration.
|
||||
2. A generative model's self-reported confidence SHALL NOT be used as authoritative extraction confidence.
|
||||
3. THE Calibration_Process SHALL evaluate isotonic, Platt, or equivalent calibration methods on held-out Gold_Corpus data.
|
||||
4. THE Pipeline SHALL report Expected Calibration Error and Brier score for probability-bearing stages.
|
||||
5. THE Router SHALL use calibrated uncertainty and explicit conflict rules to choose Fast_Path or Adjudication_Path.
|
||||
6. THE Pipeline SHALL retain stage-level confidence components for explainability.
|
||||
7. WHEN calibration data is insufficient for a class, THE Pipeline SHALL use conservative thresholds and mark the class as under-calibrated.
|
||||
|
||||
### Requirement 11: 9B Generative Adjudicator
|
||||
|
||||
**User Story:** As an analyst, I want the current reasoning capability retained for hard documents, so that specialization does not reduce intelligence on nuanced cases.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Adjudicator SHALL initially use the existing 9B-class model served by vLLM on the RTX 4070 Ti SUPER.
|
||||
2. THE Adjudicator SHALL receive selected source chunks, Evidence_Spans, candidate facts, candidate probabilities, conflicts, and a precise adjudication question rather than the entire tracked ticker list.
|
||||
3. THE Adjudicator SHALL use strict JSON Schema constrained output when supported by the endpoint.
|
||||
4. THE Adjudicator SHALL use deterministic generation settings appropriate for extraction, including a production default temperature of zero unless a benchmark proves a different value superior.
|
||||
5. THE Adjudicator SHALL NOT be asked to provide authoritative novelty, confidence, or impact values.
|
||||
6. THE Router SHALL invoke adjudication for unresolved entity aliases, contradictory evidence, multi-company causal relationships, implied consequences, complex guidance, materially incomplete fast-path results, or low calibrated confidence.
|
||||
7. THE Adjudicator SHALL return field-level decisions, evidence references, and decision reasons.
|
||||
8. IF adjudication output references evidence not supplied to it, THEN THE Verifier SHALL reject the unsupported field.
|
||||
9. THE Adjudicator SHALL remain optional for thesis prose; deterministic signal records SHALL not depend on prose generation succeeding.
|
||||
10. THE Peak GPU memory budget SHALL not exceed the measured current 9B deployment baseline by more than 5 percent unless explicitly approved.
|
||||
|
||||
### Requirement 12: Stock-Specific Impact and Horizon Model
|
||||
|
||||
**User Story:** As a trader, I want impact and horizon estimated from historical market behavior rather than language-model intuition, so that signals are tied to observed outcomes.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Pipeline SHALL separate textual sentiment from expected market impact.
|
||||
2. THE Impact_Model SHALL consume versioned features including event type probabilities, sentiment probabilities, magnitude, surprise where available, source history, novelty, company attributes, market regime, pre-event volatility, and evidence quality.
|
||||
3. THE Training_Pipeline SHALL create leakage-safe labels from abnormal returns and volume responses over configured horizons.
|
||||
4. THE Initial model family SHALL be a CPU-efficient calibrated tabular model and SHALL include a transparent deterministic baseline.
|
||||
5. THE Impact_Model SHALL output signed direction probabilities, expected magnitude, horizon probabilities, and model uncertainty.
|
||||
6. THE Production model SHALL be evaluated out-of-time and by event type, sector, market-cap bucket, and source.
|
||||
7. THE Pipeline SHALL preserve existing downstream fields through a Compatibility_Adapter while storing richer probability distributions in v3 tables.
|
||||
8. IF no trained Impact_Model is approved, THEN THE Pipeline SHALL use the deterministic baseline and SHALL NOT fall back to a generative model's impact score.
|
||||
9. THE Outcome_Evaluator SHALL feed realized outcomes back into model monitoring and retraining datasets without mutating historical predictions.
|
||||
10. THE Pipeline SHALL version feature definitions, training data ranges, model artifacts, thresholds, and calibration artifacts.
|
||||
|
||||
### Requirement 13: Versioned Intelligence Schema and Provenance
|
||||
|
||||
**User Story:** As a developer, I want a richer schema with field-level provenance, so that downstream consumers can distinguish facts, probabilities, decisions, and generated prose.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Database SHALL store v3 entities, facts, evidence spans, company signal candidates, stage runs, adjudication decisions, and model lineage in normalized or well-defined JSONB-backed tables.
|
||||
2. EVERY v3 field SHALL identify whether it is deterministic, specialist-derived, adjudicated, calibrated, or compatibility-derived.
|
||||
3. EVERY stage run SHALL record input references, output references, model versions, endpoint identity, duration, error state, and trace ID.
|
||||
4. THE Compatibility_Adapter SHALL map approved v3 outputs to existing v2 persistence records during migration.
|
||||
5. THE Compatibility_Adapter SHALL identify its own version and SHALL not overwrite original v3 probabilities.
|
||||
6. THE persisted `model_provider` and model lineage SHALL reflect the actual route used and SHALL not be hardcoded to Ollama.
|
||||
7. THE Pipeline SHALL retain raw model output only in approved object storage with configured retention and access controls.
|
||||
|
||||
### Requirement 14: Parallelism, Queues, and Resource Isolation
|
||||
|
||||
**User Story:** As an operator, I want parallel throughput without saturating the GPU or blocking unrelated stages, so that the cluster remains responsive.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Extractor SHALL support multiple in-flight documents using bounded asynchronous workers rather than a single unbounded sequential loop.
|
||||
2. THE Fast_Path and Adjudication_Path SHALL have separate queue or concurrency controls.
|
||||
3. THE Specialist_Service SHALL support dynamic batching within configured latency limits.
|
||||
4. THE Adjudicator SHALL enforce a GPU-safe concurrency semaphore coordinated with vLLM limits.
|
||||
5. THE Router SHALL apply backpressure when either path exceeds its queue-depth or latency thresholds.
|
||||
6. THE Deployment SHALL assign specialist workloads to CPU nodes and the 9B vLLM workload to the RTX 4070 Ti SUPER node by default.
|
||||
7. THE System SHALL expose queue depth, service time, batch size, GPU memory, GPU utilization, fast-path rate, and adjudication rate.
|
||||
8. WHEN the adjudicator is unavailable, THE Pipeline SHALL continue only for documents meeting a conservative fast-path acceptance threshold; all others SHALL remain queued or fail safely.
|
||||
|
||||
### Requirement 15: Observability, Audit, and Explainability
|
||||
|
||||
**User Story:** As an operator and analyst, I want to understand why a document produced a signal and which component made each decision.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Pipeline SHALL emit one distributed trace covering preprocessing, specialist stages, routing, adjudication, impact prediction, and persistence.
|
||||
2. THE Metrics SHALL include field validity, evidence coverage, entity resolution rate, sentiment agreement, calibration metrics, fast-path coverage, adjudication causes, schema failure rate, latency percentiles, token usage, and GPU-seconds per document.
|
||||
3. THE Audit API SHALL return model lineage and evidence for a document, company, and generated signal.
|
||||
4. THE UI SHALL distinguish observed facts, inferred exposure, sentiment, predicted impact, and generated narrative.
|
||||
5. THE Pipeline SHALL store routing reasons as structured codes rather than log-only text.
|
||||
6. THE Pipeline SHALL allow a reviewer to mark a field correct, incorrect, unsupported, or ambiguous and add a corrected value.
|
||||
7. Reviewer corrections SHALL be immutable audit events and SHALL feed the active-learning dataset only through an approved export process.
|
||||
|
||||
### Requirement 16: Benchmark, Shadow Mode, and Promotion Gates
|
||||
|
||||
**User Story:** As an owner, I want the new architecture proven against the current system before it affects trades, so that complexity is justified by measured improvement.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Team SHALL create a versioned Gold_Corpus covering articles, filings, press releases, transcripts, macro news, multi-company stories, contradictory reports, and long documents.
|
||||
2. THE Evaluation_Harness SHALL run the current pipeline and every proposed v3 configuration on identical inputs.
|
||||
3. THE Evaluation SHALL report field precision, recall, F1, exact-match accuracy, evidence support rate, ticker-resolution accuracy, event macro-F1, sentiment macro-F1, calibration, latency, throughput, CPU use, GPU use, and cost.
|
||||
4. THE Evaluation SHALL report results by document type, event class, source, sector, and difficulty bucket.
|
||||
5. THE Initial promotion gate SHALL require no statistically meaningful regression in any safety-critical field and measurable improvement in at least one of evidence support, calibration, schema validity, or resource efficiency.
|
||||
6. THE Initial production target SHALL achieve at least 60 percent Fast_Path coverage on the representative corpus while meeting accuracy gates.
|
||||
7. THE GPU-seconds per accepted document SHALL improve by at least 2x relative to the current 9B-every-document baseline before full promotion.
|
||||
8. THE v3 pipeline SHALL run in Shadow_Mode for a configurable period and minimum document count before it may influence aggregation.
|
||||
9. THE Promotion process SHALL support canary percentages, automatic rollback thresholds, and one-click reversion to the current pipeline.
|
||||
10. Backtest or paper-trading performance SHALL be reported separately from extraction correctness and SHALL not override failed correctness gates.
|
||||
|
||||
### Requirement 17: Active Learning and Specialist Fine-Tuning
|
||||
|
||||
**User Story:** As a model owner, I want difficult and corrected examples to improve the specialist path over time, so that fewer documents require the 9B adjudicator.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Active_Learning_Exporter SHALL select low-confidence, conflicting, adjudicated, and reviewer-corrected examples without exporting secrets or unauthorized content.
|
||||
2. THE Export format SHALL retain source text, spans, schema labels, relations, adjudicator decisions, reviewer corrections, and provenance.
|
||||
3. THE Training_Pipeline SHALL support fine-tuning the selected specialist extractor on the Stonks Oracle schema.
|
||||
4. EACH trained artifact SHALL be evaluated against a frozen holdout and the current production artifact.
|
||||
5. A specialist model SHALL not be promoted solely because it reduces adjudication rate; it SHALL also pass field-level correctness and calibration gates.
|
||||
6. THE Registry SHALL retain model cards containing training range, dataset version, intended use, limitations, and evaluation results.
|
||||
|
||||
### Requirement 18: Backward-Compatible Rollout
|
||||
|
||||
**User Story:** As a maintainer, I want to ship the new pipeline incrementally, so that existing APIs and downstream services continue operating during migration.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Current v2 extractor SHALL remain available behind a feature flag until v3 completes shadow and canary promotion.
|
||||
2. THE Compatibility_Adapter SHALL produce the fields required by aggregation, recommendation, validation, reporting, and API consumers.
|
||||
3. THE Database migration SHALL be additive before any destructive column or table change.
|
||||
4. THE Deployment SHALL permit per-agent, per-document-type, and percentage-based routing between v2 and v3.
|
||||
5. WHEN rollback is triggered, THE System SHALL route new work to v2 without deleting v3 audit data.
|
||||
6. THE Team SHALL remove deprecated provider branches, v2 prompt logic, and compatibility mappings only in a separately approved cleanup milestone.
|
||||
@@ -0,0 +1,427 @@
|
||||
# Implementation Plan: Intelligence Pipeline v3
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the monolithic 9B model extraction pipeline with a staged, evidence-grounded multi-component architecture. CPU-first specialist services handle routine extraction, sentiment, novelty, and calibration while the existing 9B vLLM model is preserved for semantic adjudication of ambiguous cases. A capability-aware inference gateway replaces duplicated provider branches, a stock-specific impact model replaces generative self-scores, and a full shadow/canary promotion process ensures measured improvement before production influence.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] 1. Rotate exposed credentials
|
||||
- [x] 1.1 Identify every live or reusable credential in `infra/helm/stonks-oracle/values-live-math.yaml` and any other tracked files
|
||||
- [x] 1.2 Rotate database, MinIO/object-store, Redis, broker, and market-data credentials
|
||||
- [x] 1.3 Disable the replaced keys and review relevant access logs
|
||||
- [x] 1.4 Remove plaintext values from the working tree without copying them into issues, PRs, logs, or spec comments
|
||||
- [x] 1.5 Purge the values from Git history using an approved coordinated history rewrite
|
||||
- [x] 1.6 Verify that old credentials no longer authenticate
|
||||
- _Requirements: 1.1, 1.2, 1.6_
|
||||
|
||||
- [x] 2. Add managed secret delivery
|
||||
- [x] 2.1 Select External Secrets, SOPS, Sealed Secrets, or the cluster-standard mechanism
|
||||
- [x] 2.2 Replace Helm secret values with secret references
|
||||
- [x] 2.3 Document bootstrap and rotation procedures
|
||||
- [x] 2.4 Add a deployment test proving pods receive required keys without values appearing in rendered manifests
|
||||
- _Requirements: 1.3, 1.6_
|
||||
|
||||
- [x] 3. Add repository secret scanning
|
||||
- [x] 3.1 Add a secret scanner to pre-commit or Kiro hooks
|
||||
- [x] 3.2 Add the scanner to pull-request and protected-branch CI
|
||||
- [x] 3.3 Add tests/fixtures that prove real-looking secrets fail and explicit safe fixtures pass
|
||||
- _Requirements: 1.4, 1.5_
|
||||
|
||||
- [x] 4. Establish the current runtime source of truth
|
||||
- [x] 4.1 Inventory active cluster deployments, agent database records, Helm releases, and environment variables
|
||||
- [x] 4.2 Record the actual model, quantization, vLLM version, max model length, max sequences, GPU utilization limit, and current provider for every agent
|
||||
- [x] 4.3 Resolve the conflicting Qwen/NuExtract defaults in code and infrastructure for the baseline branch
|
||||
- [x] 4.4 Produce `docs/intelligence-pipeline-v3/current-runtime-baseline.md` without credentials
|
||||
- _Requirements: 3.9_
|
||||
|
||||
- [x] 5. Build a baseline replay command
|
||||
- [x] 5.1 Add a CLI that replays a fixed document set through the current pipeline without writing trading outputs
|
||||
- [x] 5.2 Capture structured output, schema validity, retries, duration, token usage, GPU metrics, provider/model lineage, and current downstream mappings
|
||||
- [x] 5.3 Pin all baseline configuration and random seeds that the provider supports
|
||||
- [x] 5.4 Store baseline reports under a versioned artifact path
|
||||
- _Requirements: 16.2, 16.3_
|
||||
|
||||
- [x] 6. Define the v3 annotation schema
|
||||
- [x] 6.1 Define labels for entities, canonical companies, events, relations, numeric facts, periods, sentiment, evidence spans, direct effects, inferred exposure, and ambiguity
|
||||
- [x] 6.2 Define evidence and adjudication guidelines with positive and negative examples
|
||||
- [x] 6.3 Define which fields are safety-critical for promotion gates
|
||||
- [x] 6.4 Add schema validators and sample annotations
|
||||
- _Requirements: 16.1, 16.4_
|
||||
|
||||
- [x] 7. Create the first Gold Corpus
|
||||
- [x] 7.1 Sample at least 1,000 documents stratified by type, event class, length, source, company count, and difficulty
|
||||
- [x] 7.2 Include duplicate stories, long filings, transcripts, contradictory reports, macro events, and opposing multi-company effects
|
||||
- [x] 7.3 Double-review a hard-case subset and calculate inter-annotator agreement
|
||||
- [x] 7.4 Freeze a holdout split that cannot be used for prompt or model tuning
|
||||
- _Requirements: 16.1_
|
||||
|
||||
- [x] 8. Implement evaluation metrics
|
||||
- [x] 8.1 Implement entity/ticker precision, recall, F1, and ambiguity accuracy
|
||||
- [x] 8.2 Implement event and relation macro/micro F1
|
||||
- [x] 8.3 Implement numeric exact/tolerance-aware matching
|
||||
- [x] 8.4 Implement evidence offset validity and support rate
|
||||
- [x] 8.5 Implement sentiment macro-F1 and probability calibration metrics
|
||||
- [x] 8.6 Implement latency, throughput, token, CPU, GPU, and memory metrics
|
||||
- [x] 8.7 Generate per-document-type and per-difficulty reports
|
||||
- _Requirements: 16.3, 16.4_
|
||||
|
||||
- [x] 9. Benchmark corrected current-model extraction
|
||||
- [x] 9.1 Run the current request unchanged
|
||||
- [x] 9.2 Run the same 9B model with temperature zero
|
||||
- [x] 9.3 Run the same 9B model with strict JSON Schema output and temperature zero
|
||||
- [x] 9.4 Quantify how much of the apparent architecture gain comes from fixing the current request alone
|
||||
- _Requirements: 16.2, 16.3, 16.5_
|
||||
|
||||
- [x] 10. Add shared inference domain models
|
||||
- [x] 10.1 Create `services/shared/inference/models.py` with capabilities, target, request, result, usage, and lineage types
|
||||
- [x] 10.2 Create normalized error categories for timeout, authentication, rate limit, server, invalid response, schema, capability, and policy failures
|
||||
- [x] 10.3 Add serialization tests proving credentials and sensitive headers are excluded
|
||||
- _Requirements: 2.1, 2.8, 2.9_
|
||||
|
||||
- [x] 11. Implement `OpenAICompatibleClient`
|
||||
- [x] 11.1 Implement `/v1/chat/completions` using `httpx.AsyncClient`
|
||||
- [x] 11.2 Implement Bearer and configurable authentication headers via runtime secret resolution
|
||||
- [x] 11.3 Implement standard `response_format.json_schema` payloads
|
||||
- [x] 11.4 Implement configurable vLLM `structured_outputs` extra-body payloads
|
||||
- [x] 11.5 Implement explicit JSON-object and prompt-only fallback policies
|
||||
- [x] 11.6 Capture request ID, usage, finish reason, structured mode, retries, and provider error category
|
||||
- [x] 11.7 Revalidate parsed JSON locally against the supplied schema
|
||||
- [x] 11.8 Add contract tests against a mocked compatible server and the cluster vLLM deployment
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.7, 2.9_
|
||||
|
||||
- [x] 12. Refactor Ollama support into `OllamaNativeClient`
|
||||
- [x] 12.1 Move current Ollama request logic behind the shared request/result types
|
||||
- [x] 12.2 Honor configured max output tokens and context settings consistently
|
||||
- [x] 12.3 Preserve native schema formatting when supported and explicitly report prompt-only mode otherwise
|
||||
- [x] 12.4 Retain stall/loop detection as Ollama-specific policy without leaking it into the generic protocol
|
||||
- _Requirements: 2.1, 2.6_
|
||||
|
||||
- [x] 13. Implement capability probing
|
||||
- [x] 13.1 Probe health and model listing
|
||||
- [x] 13.2 Probe strict JSON Schema with a minimal schema
|
||||
- [x] 13.3 Probe usage metadata, seed behavior, and output-token field compatibility
|
||||
- [x] 13.4 Store probe results and software/version metadata with TTL
|
||||
- [x] 13.5 Refuse activation when declared required capabilities fail
|
||||
- _Requirements: 2.10, 2.11_
|
||||
|
||||
- [x] 14. Replace provider fallback behavior
|
||||
- [x] 14.1 Replace `VLLMClient` with an alias/profile using `OpenAICompatibleClient`
|
||||
- [x] 14.2 Make unknown providers a typed configuration error
|
||||
- [x] 14.3 Add migration warnings for `vllm` provider records
|
||||
- [x] 14.4 Add property tests proving unknown providers never invoke Ollama
|
||||
- _Requirements: 2.2, 2.6_
|
||||
|
||||
- [x] 15. Migrate all LLM consumers
|
||||
- [x] 15.1 Migrate document extraction
|
||||
- [x] 15.2 Migrate global event classification
|
||||
- [x] 15.3 Migrate thesis rewriting and remove duplicate provider branching
|
||||
- [x] 15.4 Replace direct/private `_config` mutation with an explicit target refresh or client-pool lifecycle
|
||||
- [x] 15.5 Fix persistence so actual endpoint, model, and route lineage are recorded
|
||||
- _Requirements: 2.12, 13.6_
|
||||
|
||||
- [x] 16. Add registry migrations
|
||||
- [x] 16.1 Create `inference_endpoints`
|
||||
- [x] 16.2 Create `model_deployments`
|
||||
- [x] 16.3 Create `agent_stage_bindings`
|
||||
- [x] 16.4 Add revision, audit, uniqueness, and enabled-state constraints
|
||||
- [x] 16.5 Add additive lineage columns/tables for existing performance logs
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4_
|
||||
|
||||
- [x] 17. Implement registry resolver
|
||||
- [x] 17.1 Resolve active stage bindings with TTL caching
|
||||
- [x] 17.2 Invalidate cache on revisions and failed probes
|
||||
- [x] 17.3 Resolve authentication only at invocation time
|
||||
- [x] 17.4 Add deterministic resolution and fail-closed tests
|
||||
- _Requirements: 3.5, 3.9_
|
||||
|
||||
- [x] 18. Migrate existing provider records
|
||||
- [x] 18.1 Create the current Ollama endpoint profile if in use
|
||||
- [x] 18.2 Create the current vLLM OpenAI-compatible endpoint profile
|
||||
- [x] 18.3 Create model deployments matching actual runtime state
|
||||
- [x] 18.4 Convert agent and variant provider/model fields to stage bindings while retaining compatibility reads
|
||||
- [x] 18.5 Remove conflicting runtime model defaults after migration verification
|
||||
- _Requirements: 3.8, 3.9_
|
||||
|
||||
- [x] 19. Add endpoint API and UI
|
||||
- [x] 19.1 Add CRUD endpoints that never return secret values
|
||||
- [x] 19.2 Add probe, enable, disable, and test-structured-output actions
|
||||
- [x] 19.3 Replace free-text provider inputs with protocol, endpoint, and model-deployment selectors
|
||||
- [x] 19.4 Display last probe, capabilities, model limits, and active stage bindings
|
||||
- [x] 19.5 Require confirmation for external endpoint egress enablement
|
||||
- _Requirements: 3.6, 3.7_
|
||||
|
||||
- [x] 20. Add v3 persistence tables
|
||||
- [x] 20.1 Add pipeline runs and stage runs
|
||||
- [x] 20.2 Add document chunks and evidence spans
|
||||
- [x] 20.3 Add extracted entities, facts, relations, and rejected candidates
|
||||
- [x] 20.4 Add company signal candidates and probability distributions
|
||||
- [x] 20.5 Add adjudication decisions, routing reasons, calibration references, and model lineage
|
||||
- [x] 20.6 Add idempotency and immutable-revision constraints
|
||||
- _Requirements: 13.1, 13.2, 13.3_
|
||||
|
||||
- [x] 21. Implement sentence-aware segmenter
|
||||
- [x] 21.1 Preserve source offsets and checksums
|
||||
- [x] 21.2 Add document-type-specific chunk strategies
|
||||
- [x] 21.3 Preserve filing sections and transcript speakers
|
||||
- [x] 21.4 Mark boilerplate and duplicate chunks
|
||||
- [x] 21.5 Remove the 8,000-character truncation from v3
|
||||
- [x] 21.6 Add property tests proving every chunk/evidence span maps exactly to source text
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
|
||||
|
||||
- [x] 22. Implement compatibility adapter skeleton
|
||||
- [x] 22.1 Map approved v3 records to current intelligence and impact data classes
|
||||
- [x] 22.2 Persist `hybrid` lineage plus stage details
|
||||
- [x] 22.3 Add golden mapping tests for every legacy enum and field range
|
||||
- [x] 22.4 Keep adapter output disabled outside replay/shadow mode
|
||||
- _Requirements: 13.4, 13.5, 18.2_
|
||||
|
||||
- [x] 23. Implement deterministic financial parsing
|
||||
- [x] 23.1 Parse tickers, currencies, money, percentages, basis points, ranges, EPS, revenue, dates, and fiscal periods
|
||||
- [x] 23.2 Store literal and normalized representations
|
||||
- [x] 23.3 Link each candidate to exact offsets
|
||||
- [x] 23.4 Add broad property tests for numeric formatting and unit conversions
|
||||
- _Requirements: 5.1, 5.7, 5.8_
|
||||
|
||||
- [x] 24. Integrate symbol registry resolution
|
||||
- [x] 24.1 Build canonical alias indexes from existing companies and symbol registry data
|
||||
- [x] 24.2 Return ranked candidates and ambiguity margins
|
||||
- [x] 24.3 Separate explicit mentions from inferred exposures
|
||||
- [x] 24.4 Preserve unresolved literal entities without invented tickers
|
||||
- [x] 24.5 Add tests for aliases shared by multiple companies
|
||||
- _Requirements: 5.2, 5.3, 5.4, 5.5, 5.6_
|
||||
|
||||
- [x] 25. Create specialist inference service
|
||||
- [x] 25.1 Add typed batch endpoints for entities, classification, relations, and structured extraction
|
||||
- [x] 25.2 Integrate pinned GLiNER2 Large as the initial candidate
|
||||
- [x] 25.3 Return spans, scores, model version, and schema version
|
||||
- [x] 25.4 Add bounded dynamic batching and warm-up
|
||||
- [x] 25.5 Add Kubernetes CPU deployment, health probes, and metrics
|
||||
- [x] 25.6 Add contract and load tests
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.7, 6.9_
|
||||
|
||||
- [x] 26. Integrate company-specific sentiment
|
||||
- [x] 26.1 Build company-linked evidence groups
|
||||
- [x] 26.2 Integrate pinned FinBERT baseline
|
||||
- [x] 26.3 Store full probability distributions and evidence IDs
|
||||
- [x] 26.4 Implement mixed sentiment from evidence-group disagreement
|
||||
- [x] 26.5 Benchmark and calibrate on the Gold_Corpus
|
||||
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7_
|
||||
|
||||
- [x] 27. Benchmark NuExtract 1.5 Smol
|
||||
- [x] 27.1 Add an isolated adapter and CPU/on-demand deployment
|
||||
- [x] 27.2 Test hierarchical extraction on long filings and transcripts
|
||||
- [x] 27.3 Measure incremental correctness over GLiNER2 plus deterministic parsing
|
||||
- [x] 27.4 Measure CPU latency and memory
|
||||
- [x] 27.5 Promote it only for document classes where incremental value passes a predefined gate
|
||||
- _Requirements: 6.6_
|
||||
|
||||
- [x] 28. Add evidence verification
|
||||
- [x] 28.1 Validate offsets, source text, entity association, and numeric consistency
|
||||
- [x] 28.2 Add rejected-candidate storage and reason codes
|
||||
- [x] 28.3 Benchmark a compact entailment verifier on claims exact matching cannot validate
|
||||
- [x] 28.4 Add unsupported-claim and evidence-coverage metrics
|
||||
- _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7_
|
||||
|
||||
- [x] 29. Implement retrieval-based novelty
|
||||
- [x] 29.1 Add exact and near-duplicate fingerprints
|
||||
- [x] 29.2 Add replaceable compact embedding backend
|
||||
- [x] 29.3 Index document and canonical company-event embeddings
|
||||
- [x] 29.4 Return nearest matches and similarity scores
|
||||
- [x] 29.5 Implement and version the novelty formula
|
||||
- [x] 29.6 Compare novelty values against human duplicate/novelty labels
|
||||
- _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7_
|
||||
|
||||
- [x] 30. Build confidence feature pipeline
|
||||
- [x] 30.1 Compute field-level features from extraction, resolution, evidence, sentiment, and agreement
|
||||
- [x] 30.2 Train and compare calibration methods on training folds
|
||||
- [x] 30.3 Evaluate ECE and Brier score on held-out data
|
||||
- [x] 30.4 Version and load calibration artifacts
|
||||
- [x] 30.5 Define conservative defaults for underrepresented classes
|
||||
- _Requirements: 10.1, 10.2, 10.3, 10.4, 10.6, 10.7_
|
||||
|
||||
- [x] 31. Implement deterministic routing engine
|
||||
- [x] 31.1 Define routing reason enums
|
||||
- [x] 31.2 Implement hard ambiguity/conflict rules
|
||||
- [x] 31.3 Implement calibrated fast-path thresholds by document and event type
|
||||
- [x] 31.4 Store every route decision and feature snapshot
|
||||
- [x] 31.5 Add property tests for determinism and threshold boundaries
|
||||
- _Requirements: 10.5, 11.6_
|
||||
|
||||
- [x] 32. Define adjudication schemas
|
||||
- [x] 32.1 Define candidate, conflict, question, evidence, and decision models
|
||||
- [x] 32.2 Exclude authoritative confidence, novelty, impact, and horizon from the model output
|
||||
- [x] 32.3 Require evidence IDs for every material decision
|
||||
- _Requirements: 11.2, 11.5, 11.7_
|
||||
|
||||
- [x] 33. Build focused adjudication prompts
|
||||
- [x] 33.1 Build packets from only relevant chunks and candidates
|
||||
- [x] 33.2 Use strict JSON Schema and temperature zero
|
||||
- [x] 33.3 Set a bounded output budget appropriate to decisions rather than long summaries
|
||||
- [x] 33.4 Add prompt/version metadata and exact provider lineage
|
||||
- _Requirements: 11.2, 11.3, 11.4_
|
||||
|
||||
- [x] 34. Deploy and validate the 9B adjudicator
|
||||
- [x] 34.1 Pin the approved 9B model and vLLM version
|
||||
- [x] 34.2 Verify strict structured output with the deployment's actual vLLM version
|
||||
- [x] 34.3 Measure peak VRAM against the current baseline and enforce the +5 percent gate
|
||||
- [x] 34.4 Load-test concurrency and select a safe application semaphore
|
||||
- [x] 34.5 Add availability and queue-depth alerts
|
||||
- _Requirements: 11.1, 11.10_
|
||||
|
||||
- [x] 35. Add post-adjudication verification
|
||||
- [x] 35.1 Verify every referenced evidence ID was included in the packet
|
||||
- [x] 35.2 Reject unsupported or schema-incompatible decisions
|
||||
- [x] 35.3 Preserve both pre-adjudication candidates and final decisions
|
||||
- [x] 35.4 Route repeated failures to review rather than accepting repaired defaults
|
||||
- _Requirements: 11.8, 8.7_
|
||||
|
||||
- [x] 36. Define event-time feature snapshots
|
||||
- [x] 36.1 Define feature names, types, timing rules, and missing-value policy
|
||||
- [x] 36.2 Include event, sentiment, magnitude, surprise, source, novelty, evidence, company, volatility, volume, and regime features
|
||||
- [x] 36.3 Persist immutable feature snapshots at prediction time
|
||||
- [x] 36.4 Add leakage tests preventing post-event data from entering features
|
||||
- _Requirements: 12.2, 12.10_
|
||||
|
||||
- [x] 37. Build outcome labels
|
||||
- [x] 37.1 Define approved market benchmarks and abnormal-return calculations
|
||||
- [x] 37.2 Generate signed and absolute response labels for intraday, 1d, 7d, 30d, and 90d horizons
|
||||
- [x] 37.3 Generate abnormal-volume and time-to-peak labels where data quality permits
|
||||
- [x] 37.4 Version label-generation code and market-data snapshots
|
||||
- _Requirements: 12.3_
|
||||
|
||||
- [x] 38. Implement deterministic impact baseline
|
||||
- [x] 38.1 Map event classes, sentiment, magnitude, evidence, novelty, and source credibility to conservative outputs
|
||||
- [x] 38.2 Unit-test every event type and boundary
|
||||
- [x] 38.3 Use this baseline whenever no approved trained model exists
|
||||
- _Requirements: 12.4, 12.8_
|
||||
|
||||
- [x] 39. Train calibrated tabular impact models
|
||||
- [x] 39.1 Train CPU-efficient gradient-boosted candidates for direction, magnitude, and horizon
|
||||
- [x] 39.2 Use walk-forward/out-of-time splits
|
||||
- [x] 39.3 Calibrate probabilities on a separate calibration fold
|
||||
- [x] 39.4 Report metrics by event, sector, market cap, source, and regime
|
||||
- [x] 39.5 Register artifacts, feature versions, training ranges, and model cards
|
||||
- _Requirements: 12.4, 12.5, 12.6, 12.10_
|
||||
|
||||
- [x] 40. Integrate impact outputs
|
||||
- [x] 40.1 Store full direction, magnitude, horizon, and uncertainty outputs
|
||||
- [x] 40.2 Map approved outputs to legacy `impact_score` and `impact_horizon` through the compatibility adapter
|
||||
- [x] 40.3 Remove generative impact/novelty/confidence from aggregation inputs in v3 mode
|
||||
- [x] 40.4 Add comparison dashboards against realized outcomes
|
||||
- _Requirements: 12.1, 12.7, 12.9_
|
||||
|
||||
- [x] 41. Implement v3 pipeline orchestrator
|
||||
- [x] 41.1 Create explicit stage state transitions and idempotency keys
|
||||
- [x] 41.2 Add fast-path, adjudication, persistence, and review queues
|
||||
- [x] 41.3 Implement leases, retry policies, dead-letter handling, and resumable stages
|
||||
- [x] 41.4 Keep v2 and v3 routing behind independent feature flags
|
||||
- _Requirements: 14.1, 14.2, 14.5, 14.8_
|
||||
|
||||
- [x] 42. Add bounded application parallelism
|
||||
- [x] 42.1 Replace the single sequential extraction loop for v3 with configurable async workers
|
||||
- [x] 42.2 Add specialist micro-batching
|
||||
- [x] 42.3 Add adjudicator semaphore and queue backpressure
|
||||
- [x] 42.4 Add load shedding rules that never drop safety-critical documents silently
|
||||
- _Requirements: 14.1, 14.3, 14.4, 14.5_
|
||||
|
||||
- [x] 43. Add traces and metrics
|
||||
- [x] 43.1 Trace every stage under one document trace ID
|
||||
- [x] 43.2 Add stage latency, errors, batch size, queue depth, and route metrics
|
||||
- [x] 43.3 Add field accuracy, evidence coverage, calibration, fast-path rate, and adjudication reason dashboards
|
||||
- [x] 43.4 Add GPU memory, utilization, and GPU-seconds per document
|
||||
- [x] 43.5 Add alerts for schema failures, unsupported claims, calibration drift, queue saturation, and provider probe failures
|
||||
- _Requirements: 14.7, 15.1, 15.2, 15.5_
|
||||
|
||||
- [x] 44. Add audit/review API and UI
|
||||
- [x] 44.1 Display source evidence and offsets for each fact
|
||||
- [x] 44.2 Display specialist probabilities, routing reasons, adjudicator decisions, and impact-model outputs separately
|
||||
- [x] 44.3 Allow immutable reviewer correction events
|
||||
- [x] 44.4 Add filters for low confidence, unsupported claims, and adjudicated documents
|
||||
- _Requirements: 15.3, 15.4, 15.6, 15.7_
|
||||
|
||||
- [x] 45. Run offline replay
|
||||
- [x] 45.1 Compare every required system configuration on the Gold_Corpus
|
||||
- [x] 45.2 Publish field-level, calibration, resource, and difficulty-bucket reports
|
||||
- [x] 45.3 Confirm corrected current-9B baseline versus full v3 incremental gain
|
||||
- [x] 45.4 Reject or retune any stage failing safety-critical gates
|
||||
- _Requirements: 16.2, 16.3, 16.4, 16.5_
|
||||
|
||||
- [x] 46. Enable production shadow mode
|
||||
- [x] 46.1 Run v3 for live documents without affecting aggregation or trading
|
||||
- [x] 46.2 Compare v2/v3 disagreements and sample reviews by risk
|
||||
- [x] 46.3 Measure fast-path coverage, GPU reduction, and operational stability
|
||||
- [x] 46.4 Require the configured minimum shadow duration and document count
|
||||
- _Requirements: 16.6, 16.7, 16.8_
|
||||
|
||||
- [x] 47. Canary compatibility outputs
|
||||
- [x] 47.1 Enable v3 adapter outputs for non-trading consumers first
|
||||
- [x] 47.2 Add percentage- and document-type-based routing
|
||||
- [x] 47.3 Configure automatic rollback on correctness, latency, queue, or availability thresholds
|
||||
- [x] 47.4 Verify rollback leaves v3 audit records intact
|
||||
- _Requirements: 16.9, 18.4, 18.5_
|
||||
|
||||
- [x] 48. Canary signal influence
|
||||
- [x] 48.1 Enable v3 signals in paper trading at a small percentage
|
||||
- [x] 48.2 Report extraction correctness separately from trading outcomes
|
||||
- [x] 48.3 Review material recommendation divergences
|
||||
- [x] 48.4 Promote only after explicit owner approval and all gates pass
|
||||
- _Requirements: 16.9, 16.10_
|
||||
|
||||
- [x] 49. Build active-learning export
|
||||
- [x] 49.1 Select low-confidence, conflicting, adjudicated, and corrected cases
|
||||
- [x] 49.2 Remove or policy-filter sensitive content
|
||||
- [x] 49.3 Export source spans, labels, relations, decisions, and provenance in a versioned format
|
||||
- _Requirements: 17.1, 17.2_
|
||||
|
||||
- [x] 50. Fine-tune specialist extractor
|
||||
- [x] 50.1 Train GLiNER2 candidate artifacts on the Stonks Oracle schema
|
||||
- [x] 50.2 Evaluate against frozen holdout and production artifact
|
||||
- [x] 50.3 Calibrate new scores and update routing thresholds
|
||||
- [x] 50.4 Promote only when correctness gates pass, not merely when adjudication rate falls
|
||||
- _Requirements: 17.3, 17.4, 17.5, 17.6_
|
||||
|
||||
- [x] 51. Deprecate legacy paths
|
||||
- [x] 51.1 Remove duplicated `VLLMClient`/provider branching after all consumers use the gateway
|
||||
- [x] 51.2 Remove v2 8,000-character truncation and monolithic extraction prompt after v2 retirement
|
||||
- [x] 51.3 Remove obsolete environment/model defaults and provider free-text fields
|
||||
- [x] 51.4 Remove compatibility adapter only after every downstream consumer reads v3 natively
|
||||
- [x] 51.5 Archive final migration and benchmark reports
|
||||
- _Requirements: 18.6_
|
||||
|
||||
- [x] 52. Checkpoint — Ensure all tests pass
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
## Task Dependency Graph
|
||||
|
||||
```json
|
||||
{
|
||||
"waves": [
|
||||
{ "id": 0, "tasks": ["1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "2.1", "2.2", "2.3", "2.4", "3.1", "3.2", "3.3", "4.1", "4.2", "4.3", "4.4", "5.1", "5.2", "5.3", "5.4"] },
|
||||
{ "id": 1, "tasks": ["6.1", "6.2", "6.3", "6.4", "7.1", "7.2", "7.3", "7.4", "8.1", "8.2", "8.3", "8.4", "8.5", "8.6", "8.7", "9.1", "9.2", "9.3", "9.4"] },
|
||||
{ "id": 2, "tasks": ["10.1", "10.2", "10.3", "11.1", "11.2", "11.3", "11.4", "11.5", "11.6", "11.7", "11.8", "12.1", "12.2", "12.3", "12.4"] },
|
||||
{ "id": 3, "tasks": ["13.1", "13.2", "13.3", "13.4", "13.5", "14.1", "14.2", "14.3", "14.4", "15.1", "15.2", "15.3", "15.4", "15.5", "16.1", "16.2", "16.3", "16.4", "16.5"] },
|
||||
{ "id": 4, "tasks": ["17.1", "17.2", "17.3", "17.4", "18.1", "18.2", "18.3", "18.4", "18.5", "19.1", "19.2", "19.3", "19.4", "19.5", "20.1", "20.2", "20.3", "20.4", "20.5", "20.6", "21.1", "21.2", "21.3", "21.4", "21.5", "21.6", "22.1", "22.2", "22.3", "22.4"] },
|
||||
{ "id": 5, "tasks": ["23.1", "23.2", "23.3", "23.4", "24.1", "24.2", "24.3", "24.4", "24.5", "25.1", "25.2", "25.3", "25.4", "25.5", "25.6", "26.1", "26.2", "26.3", "26.4", "26.5", "27.1", "27.2", "27.3", "27.4", "27.5", "28.1", "28.2", "28.3", "28.4"] },
|
||||
{ "id": 6, "tasks": ["29.1", "29.2", "29.3", "29.4", "29.5", "29.6", "30.1", "30.2", "30.3", "30.4", "30.5", "31.1", "31.2", "31.3", "31.4", "31.5"] },
|
||||
{ "id": 7, "tasks": ["32.1", "32.2", "32.3", "33.1", "33.2", "33.3", "33.4", "34.1", "34.2", "34.3", "34.4", "34.5", "35.1", "35.2", "35.3", "35.4"] },
|
||||
{ "id": 8, "tasks": ["36.1", "36.2", "36.3", "36.4", "37.1", "37.2", "37.3", "37.4", "38.1", "38.2", "38.3", "39.1", "39.2", "39.3", "39.4", "39.5", "40.1", "40.2", "40.3", "40.4"] },
|
||||
{ "id": 9, "tasks": ["41.1", "41.2", "41.3", "41.4", "42.1", "42.2", "42.3", "42.4", "43.1", "43.2", "43.3", "43.4", "43.5", "44.1", "44.2", "44.3", "44.4"] },
|
||||
{ "id": 10, "tasks": ["45.1", "45.2", "45.3", "45.4", "46.1", "46.2", "46.3", "46.4", "47.1", "47.2", "47.3", "47.4", "48.1", "48.2", "48.3", "48.4"] },
|
||||
{ "id": 11, "tasks": ["49.1", "49.2", "49.3", "50.1", "50.2", "50.3", "50.4", "51.1", "51.2", "51.3", "51.4", "51.5"] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- This plan is intentionally staged so the current system remains available until the replacement is measured and promoted
|
||||
- Task 1 (credential rotation) is a **BLOCKER** — must complete before any feature deployment
|
||||
- Tasks within the same wave may run in parallel; tasks in later waves depend on earlier waves completing
|
||||
- Each phase ends with an explicit evidence artifact: test output, benchmark report, migration result, or deployment probe
|
||||
- The current v2 extractor remains available behind a feature flag until v3 completes shadow and canary promotion
|
||||
- Peak GPU memory must not exceed the measured current 9B deployment baseline by more than 5 percent
|
||||
- The definition of done requires: credentials rotated, all consumers on shared gateway, evidence-linked outputs, calibrated scores replacing generative self-scores, 9B preserved for adjudication, and shadow/canary gates passed
|
||||
- Rollback to the current production path must be exercised successfully before full promotion
|
||||
- Property tests validate deterministic behaviors (unknown providers fail closed, chunk offsets map to source, routing thresholds are deterministic)
|
||||
- Deprecated legacy paths (task 51) require separate approval and must not proceed until all downstream consumers read v3 natively
|
||||
@@ -0,0 +1 @@
|
||||
{"specId": "f5d99301-94ef-4dc2-8ba4-ccefeee7ecba", "workflowType": "requirements-first", "specType": "bugfix"}
|
||||
@@ -0,0 +1,73 @@
|
||||
# Bugfix Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
Multiple operational bugs discovered in the stonks-beta namespace prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput. The core issue is that the outcome evaluation → metrics computation → quality gate pipeline is completely disconnected from the production scheduler, making the platform unable to self-calibrate or validate predictions. Additionally, Polygon API rate limiting causes ~40% request failures per cycle, a broken config query prevents the v3 engine from being toggled, several periodic snapshot tasks are missing from the scheduler, the lake-publisher deployment is idle/redundant, and order rejection reasons are lost.
|
||||
|
||||
## Bug Analysis
|
||||
|
||||
### Current Behavior (Defect)
|
||||
|
||||
1.1 WHEN the scheduler enqueues ingestion jobs for all 50 tickers' news_api and market_api sources simultaneously THEN the system exhausts the Polygon free-tier rate limit (5 req/min) resulting in ~40% of sources receiving HTTP 429 Too Many Requests every cycle
|
||||
|
||||
1.2 WHEN the aggregation worker reads the v3_engine_enabled flag via `_V3_ENGINE_FLAG_QUERY` THEN the system queries non-existent columns `key` and `value` on the `risk_configs` table (actual schema: `name` varchar, `config` JSONB) causing a PostgreSQL error every aggregation cycle
|
||||
|
||||
1.3 WHEN a scheduler cycle completes THEN the system never calls `evaluate_matured_predictions()` because it is not wired into the scheduler's main loop — only imported in `backtest_replay.py`
|
||||
|
||||
1.4 WHEN a scheduler cycle completes THEN the system never calls `compute_and_store_metric_snapshots()` because it is not wired into the scheduler's main loop — only called from backtest replay
|
||||
|
||||
1.5 WHEN the model quality gate evaluates trading eligibility THEN the system always fails with "no model metric snapshot available — defaulting to paper-only" because `model_metric_snapshots` table is permanently empty (consequence of bug 1.4)
|
||||
|
||||
1.6 WHEN the trading engine runs daily THEN the system never captures portfolio state snapshots to the `portfolio_snapshots` table because no periodic scheduler task invokes this capture
|
||||
|
||||
1.7 WHEN the trading engine runs daily THEN the system never captures risk state snapshots to the `daily_risk_snapshots` table because no periodic scheduler task invokes this capture
|
||||
|
||||
1.8 WHEN a prediction snapshot is created while Polygon rate-limiting has prevented the market data fetch THEN the system stores NULL in `price_at_prediction` (affecting 21% of snapshots), degrading downstream outcome evaluation accuracy
|
||||
|
||||
1.9 WHEN the standalone `lake-publisher` deployment polls `stonks:beta:queue:lake_publish` THEN the queue is always empty (0 items) because all lake publishing happens inline in broker-adapter and recommendation services — the deployment consumes zero work and wastes resources
|
||||
|
||||
1.10 WHEN Alpaca returns HTTP 401 for an order submission THEN the system sets order status to "rejected" but leaves the `rejection_reason` column NULL, capturing the error message only in the `decision_trace` JSONB field
|
||||
|
||||
### Expected Behavior (Correct)
|
||||
|
||||
2.1 WHEN the scheduler enqueues ingestion jobs for Polygon-backed sources (news_api, market_api) THEN the system SHALL pace/stagger requests across the polling interval to stay within the Polygon rate limit, achieving near-zero 429 responses per cycle
|
||||
|
||||
2.2 WHEN the aggregation worker reads the v3_engine_enabled flag THEN the system SHALL query `SELECT config FROM risk_configs WHERE name = 'v3_engine_enabled'` and parse the JSONB value to determine the boolean toggle state
|
||||
|
||||
2.3 WHEN a scheduler cycle completes and sufficient time has elapsed since the last evaluation THEN the system SHALL call `evaluate_matured_predictions()` to evaluate prediction snapshots whose horizon has elapsed, populating the `prediction_outcomes` table
|
||||
|
||||
2.4 WHEN a scheduler cycle completes and sufficient time has elapsed since the last computation THEN the system SHALL call `compute_and_store_metric_snapshots()` to compute aggregate model metrics across all lookback/horizon combinations, populating `model_metric_snapshots`
|
||||
|
||||
2.5 WHEN the model quality gate evaluates trading eligibility THEN the system SHALL have recent metric snapshots available and evaluate thresholds against actual model performance data
|
||||
|
||||
2.6 WHEN market hours close (or on a daily schedule) THEN the system SHALL capture and persist the current portfolio state to `portfolio_snapshots` including value, returns, positions, and risk metrics
|
||||
|
||||
2.7 WHEN market hours close (or on a daily schedule) THEN the system SHALL capture and persist the current risk state to `daily_risk_snapshots` including portfolio value, daily P&L, trade count, and sector positions
|
||||
|
||||
2.8 WHEN a prediction snapshot is created and market price is unavailable due to rate limiting THEN the system SHALL retry the price fetch or defer the snapshot until price data is available, reducing NULL `price_at_prediction` occurrences to near zero
|
||||
|
||||
2.9 WHEN the lake-publisher deployment architecture is reviewed THEN the system SHALL either route lake publish jobs through the Redis queue to the standalone deployment, or remove the redundant deployment — eliminating the idle pod
|
||||
|
||||
2.10 WHEN Alpaca returns an HTTP error (401, 403, or any rejection) for an order submission THEN the system SHALL populate the `rejection_reason` column with the HTTP error message/status in addition to recording it in `decision_trace`
|
||||
|
||||
### Unchanged Behavior (Regression Prevention)
|
||||
|
||||
3.1 WHEN sources with valid rate-limit headroom are enqueued THEN the system SHALL CONTINUE TO enqueue and process them without artificial delay
|
||||
|
||||
3.2 WHEN risk_configs is queried for other configuration keys (e.g., `model_quality_gate_config`, `macro_enabled`) THEN the system SHALL CONTINUE TO read them correctly using the existing `name`/`config` column pattern
|
||||
|
||||
3.3 WHEN the backtest replay module calls `evaluate_matured_predictions()` and `compute_and_store_metric_snapshots()` THEN the system SHALL CONTINUE TO execute them as part of backtest validation
|
||||
|
||||
3.4 WHEN prediction snapshots are created with available market prices THEN the system SHALL CONTINUE TO store the correct `price_at_prediction` value immediately
|
||||
|
||||
3.5 WHEN the existing inline lake publishing in broker-adapter and recommendation services writes facts THEN the system SHALL CONTINUE TO produce correct Parquet partitions in MinIO
|
||||
|
||||
3.6 WHEN orders succeed (HTTP 200 from Alpaca) THEN the system SHALL CONTINUE TO process them normally without modifying the `rejection_reason` column
|
||||
|
||||
3.7 WHEN the scheduler runs ingestion, extraction, aggregation, recommendation, and trading tasks THEN the system SHALL CONTINUE TO execute them on the existing cadence without disruption
|
||||
|
||||
3.8 WHEN the trading engine makes decisions and submits orders THEN the system SHALL CONTINUE TO record full decision context in `decision_trace` JSONB as before
|
||||
|
||||
3.9 WHEN the model quality gate passes (once metrics are populated) THEN the system SHALL CONTINUE TO allow promotion to live trading mode per existing threshold logic
|
||||
|
||||
3.10 WHEN the reporting collector fetches portfolio_snapshots and daily_risk_snapshots for report generation THEN the system SHALL CONTINUE TO query and render them using the existing schema
|
||||
@@ -0,0 +1,396 @@
|
||||
# Technical Design: ops-pipeline-fixes
|
||||
|
||||
## Overview
|
||||
|
||||
This design addresses 10 operational bugs that prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput in the `stonks-beta` namespace. The fixes span the scheduler (rate limiting + periodic tasks), aggregation worker (config query), broker service (rejection reason), prediction snapshot (price fallback), and Helm chart (dead pod removal). All changes are localized with graceful fallbacks and no schema migrations required.
|
||||
|
||||
## Bug Details
|
||||
|
||||
Multiple operational bugs in the `stonks-beta` namespace prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput. The core pipeline (ingestion → extraction → aggregation → recommendation → trading) flows end-to-end, but:
|
||||
- The outcome evaluation → metrics computation → quality gate feedback loop is completely disconnected
|
||||
- Polygon API rate limiting causes ~40% ingestion failures per cycle
|
||||
- A broken config query prevents the v3 engine toggle from working
|
||||
- Portfolio/risk snapshots are never captured
|
||||
- Order rejection reasons are lost
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Scheduler (services/scheduler/app.py)"
|
||||
A[schedule_cycle] -->|paced enqueue| B[Ingestion Queue]
|
||||
C[validation_cycle] -->|hourly| D[evaluate_matured_predictions]
|
||||
C -->|after outcomes| E[compute_and_store_metric_snapshots]
|
||||
F[snapshot_cycle] -->|daily 16:30 ET| G[capture_portfolio_snapshot]
|
||||
F -->|daily 16:30 ET| H[capture_risk_snapshot]
|
||||
end
|
||||
|
||||
subgraph "Aggregation (services/aggregation/worker.py)"
|
||||
I[_read_v3_flag] -->|fixed query| J[risk_configs.config JSONB]
|
||||
end
|
||||
|
||||
subgraph "Broker (services/adapters/broker_service.py)"
|
||||
K[persist_order] -->|rejected status| L[orders.rejection_reason]
|
||||
end
|
||||
|
||||
D --> O[prediction_outcomes]
|
||||
E --> P[model_metric_snapshots]
|
||||
P --> Q[Quality Gate]
|
||||
```
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
2.1 The scheduler SHALL pace Polygon API requests within the free-tier limit (~5 req/min), achieving near-zero 429 responses per cycle.
|
||||
|
||||
2.2 The aggregation worker SHALL read `v3_engine_enabled` from the `risk_configs` JSONB `config` column (not non-existent `key`/`value` columns).
|
||||
|
||||
2.3 The scheduler SHALL call `evaluate_matured_predictions()` hourly to populate `prediction_outcomes`.
|
||||
|
||||
2.4 The scheduler SHALL call `compute_and_store_metric_snapshots()` after outcome evaluation to populate `model_metric_snapshots`.
|
||||
|
||||
2.5 The quality gate SHALL have recent metric data available once the validation cycle runs.
|
||||
|
||||
2.6 The scheduler SHALL capture daily portfolio snapshots to `portfolio_snapshots` after market close.
|
||||
|
||||
2.7 The scheduler SHALL capture daily risk snapshots to `daily_risk_snapshots` after market close.
|
||||
|
||||
2.8 Prediction snapshots SHALL fall back to positions table prices when market_snapshots data is unavailable.
|
||||
|
||||
2.9 The lake-publisher deployment SHALL be scaled to 0 (idle pod, wasted resources).
|
||||
|
||||
2.10 The broker service SHALL populate `rejection_reason` on orders when broker or risk engine rejects.
|
||||
|
||||
## Hypothesized Root Cause
|
||||
|
||||
### Bug 1.1 — Polygon Rate Limiting
|
||||
`POLYGON_GLOBAL_RATE_LIMIT = 45` in `services/scheduler/app.py` is set for a paid Polygon plan but the deployed instance uses the free tier (5 req/min). All 50+ sources are attempted per cycle, exhausting the limit instantly.
|
||||
|
||||
### Bug 1.2 — v3_engine_enabled Config Read
|
||||
`_V3_ENGINE_FLAG_QUERY` in `services/aggregation/worker.py` reads `SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'`. The actual table has columns `name` (varchar) and `config` (JSONB) — no `key` or `value` column exists.
|
||||
|
||||
### Bug 1.3 & 1.4 — Outcome Evaluator & Metrics Never Scheduled
|
||||
`evaluate_matured_predictions()` and `compute_and_store_metric_snapshots()` exist in `services/validation/` but are only imported in `services/trading/backtest_replay.py`. The scheduler main loop in `services/scheduler/app.py` has no call to either function.
|
||||
|
||||
### Bug 1.5 — Quality Gate Permanently Failing
|
||||
`services/trading/model_quality_gate.py` queries `model_metric_snapshots` which is always empty (consequence of 1.4). Returns "no model metric snapshot available — defaulting to paper-only" every time.
|
||||
|
||||
### Bug 1.6 & 1.7 — Portfolio/Risk Snapshots
|
||||
The trading engine has `_persist_daily_snapshot()` but it only executes when the engine's main loop is actively processing trades. The trading-engine pod shows only health checks — its main loop isn't cycling because there are no active trade triggers flowing through it. No fallback capture exists in the scheduler.
|
||||
|
||||
### Bug 1.8 — Market Price Gaps
|
||||
`services/validation/prediction_snapshot.py` queries `market_snapshots` for price at prediction time. When Polygon rate limiting prevents market data fetches, no snapshot exists and `price_at_prediction` is NULL. 21% of snapshots affected.
|
||||
|
||||
### Bug 1.9 — Lake Publisher Idle
|
||||
The standalone `lake-publisher` deployment polls `stonks:beta:queue:lake_publish` but all services (broker-adapter, recommendation) import `services.lake_publisher.worker` directly and publish inline — never pushing to the Redis queue.
|
||||
|
||||
### Bug 1.10 — Order rejection_reason NULL
|
||||
`_INSERT_ORDER` SQL in `services/adapters/broker_service.py` doesn't include `rejection_reason` or `rejected_at` columns. The error is stored in `decision_trace` JSONB but the dedicated column stays NULL. The reconciliation path (`_reconcile_open_orders`) does set these columns, but initial persist does not.
|
||||
|
||||
## Fix Implementation
|
||||
|
||||
### Fix 1: Polygon Rate Limit Constant (Bug 1.1)
|
||||
|
||||
**File:** `services/scheduler/app.py`
|
||||
|
||||
Replace the hardcoded constant with an env-configurable value defaulting to 5:
|
||||
|
||||
```python
|
||||
# Before:
|
||||
POLYGON_GLOBAL_RATE_LIMIT: int = 45
|
||||
|
||||
# After:
|
||||
POLYGON_GLOBAL_RATE_LIMIT: int = int(os.getenv("POLYGON_GLOBAL_RATE_LIMIT", "5"))
|
||||
```
|
||||
|
||||
The existing `check_rate_limit()` function already implements per-minute windowed counting and skips sources once the limit is hit. By reducing the constant to match the free-tier limit, the system will naturally pace — enqueuing ~5 Polygon sources per minute across scheduler ticks (15s interval = 4 ticks/min). Skipped sources are retried next cycle.
|
||||
|
||||
**Validates:** Bugfix 2.1; Regression 3.1, 3.7
|
||||
|
||||
---
|
||||
|
||||
### Fix 2: v3_engine_enabled Config Query (Bug 1.2)
|
||||
|
||||
**File:** `services/aggregation/worker.py`
|
||||
|
||||
Replace the broken query and function:
|
||||
|
||||
```python
|
||||
# Before:
|
||||
_V3_ENGINE_FLAG_QUERY = """
|
||||
SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'
|
||||
"""
|
||||
|
||||
# After:
|
||||
_V3_ENGINE_FLAG_QUERY = """
|
||||
SELECT config->>'v3_engine_enabled' AS enabled
|
||||
FROM risk_configs
|
||||
WHERE name = 'default' AND active = TRUE
|
||||
LIMIT 1
|
||||
"""
|
||||
|
||||
async def _read_v3_flag(pool: asyncpg.Pool) -> bool:
|
||||
"""Read v3_engine_enabled from risk_configs JSONB. Default False on error."""
|
||||
try:
|
||||
row = await pool.fetchrow(_V3_ENGINE_FLAG_QUERY)
|
||||
if row and row["enabled"]:
|
||||
return row["enabled"].lower() in ("true", "1", "yes")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning("Failed to read v3_engine_enabled flag: %s", e)
|
||||
return False
|
||||
```
|
||||
|
||||
Reads from the `default` active risk_config's JSONB `config` field. Falls back to False (unchanged fail-safe).
|
||||
|
||||
**Validates:** Bugfix 2.2; Regression 3.2
|
||||
|
||||
---
|
||||
|
||||
### Fix 3: Validation Cycle in Scheduler (Bugs 1.3, 1.4, 1.5)
|
||||
|
||||
**File:** `services/scheduler/app.py`
|
||||
|
||||
Add a new periodic task (every ~240 ticks = ~60 minutes):
|
||||
|
||||
```python
|
||||
# New constant:
|
||||
VALIDATION_CYCLE_INTERVAL = int(os.getenv("VALIDATION_CYCLE_INTERVAL", "240"))
|
||||
|
||||
# New counter in main():
|
||||
validation_counter = 0
|
||||
|
||||
# In main loop after existing periodic tasks:
|
||||
validation_counter += 1
|
||||
if validation_counter >= VALIDATION_CYCLE_INTERVAL:
|
||||
validation_counter = 0
|
||||
await run_validation_cycle(pool)
|
||||
```
|
||||
|
||||
New function:
|
||||
|
||||
```python
|
||||
async def run_validation_cycle(pool: asyncpg.Pool) -> None:
|
||||
"""Run outcome evaluation and metric computation (hourly).
|
||||
|
||||
Requirements: 2.3, 2.4, 2.5
|
||||
"""
|
||||
from services.validation.outcome_evaluator import evaluate_matured_predictions
|
||||
from services.validation.metrics import compute_and_store_metric_snapshots
|
||||
|
||||
try:
|
||||
outcomes = await evaluate_matured_predictions(pool)
|
||||
logger.info("Validation: evaluated %d prediction outcomes", outcomes)
|
||||
except Exception:
|
||||
logger.exception("Validation: outcome evaluation failed")
|
||||
return # Skip metrics if outcomes failed
|
||||
|
||||
try:
|
||||
snapshots = await compute_and_store_metric_snapshots(pool)
|
||||
logger.info("Validation: computed %d metric snapshots", len(snapshots))
|
||||
except Exception:
|
||||
logger.exception("Validation: metric computation failed")
|
||||
```
|
||||
|
||||
**Validates:** Bugfix 2.3, 2.4, 2.5; Regression 3.3
|
||||
|
||||
---
|
||||
|
||||
### Fix 4: Daily Portfolio & Risk Snapshots (Bugs 1.6, 1.7)
|
||||
|
||||
**File:** `services/scheduler/app.py`
|
||||
|
||||
Add a daily snapshot task that runs every ~60 minutes but only captures once per day after 16:30 ET:
|
||||
|
||||
```python
|
||||
SNAPSHOT_CYCLE_INTERVAL = int(os.getenv("SNAPSHOT_CYCLE_INTERVAL", "240"))
|
||||
snapshot_counter = 0
|
||||
|
||||
# In main loop:
|
||||
snapshot_counter += 1
|
||||
if snapshot_counter >= SNAPSHOT_CYCLE_INTERVAL:
|
||||
snapshot_counter = 0
|
||||
await maybe_capture_daily_snapshots(pool)
|
||||
```
|
||||
|
||||
New function:
|
||||
|
||||
```python
|
||||
async def maybe_capture_daily_snapshots(pool: asyncpg.Pool) -> None:
|
||||
"""Capture portfolio and risk snapshots once daily after market close.
|
||||
|
||||
Requirements: 2.6, 2.7
|
||||
"""
|
||||
et_now = datetime.now(ZoneInfo("America/New_York"))
|
||||
|
||||
# Only after 4:30 PM ET
|
||||
if et_now.hour < 16 or (et_now.hour == 16 and et_now.minute < 30):
|
||||
return
|
||||
|
||||
today = et_now.date()
|
||||
|
||||
# Already captured today?
|
||||
existing = await pool.fetchval(
|
||||
"SELECT 1 FROM portfolio_snapshots WHERE snapshot_date = $1 LIMIT 1",
|
||||
today,
|
||||
)
|
||||
if existing:
|
||||
return
|
||||
|
||||
# Portfolio snapshot from positions + account data
|
||||
try:
|
||||
positions = await pool.fetch("SELECT * FROM positions WHERE quantity > 0")
|
||||
portfolio_value = sum(
|
||||
float(r["current_price"] or 0) * float(r["quantity"])
|
||||
for r in positions
|
||||
)
|
||||
unrealized_pnl = sum(float(r["unrealized_pnl"] or 0) for r in positions)
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO portfolio_snapshots
|
||||
(snapshot_date, portfolio_value, unrealized_pnl, positions)
|
||||
VALUES ($1, $2, $3, $4::jsonb)""",
|
||||
today, portfolio_value, unrealized_pnl,
|
||||
json.dumps([dict(r) for r in positions], default=str),
|
||||
)
|
||||
logger.info("Captured portfolio snapshot: value=%.2f", portfolio_value)
|
||||
except Exception:
|
||||
logger.exception("Failed to capture portfolio snapshot")
|
||||
|
||||
# Risk snapshot from daily activity
|
||||
try:
|
||||
daily_orders = await pool.fetchval(
|
||||
"SELECT count(*) FROM orders WHERE created_at::date = $1", today
|
||||
)
|
||||
daily_pnl = sum(float(r["unrealized_pnl"] or 0) for r in positions) if positions else 0.0
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO daily_risk_snapshots
|
||||
(account_id, snapshot_date, portfolio_value, daily_pnl, daily_trade_count)
|
||||
VALUES ((SELECT id FROM broker_accounts LIMIT 1), $1, $2, $3, $4)
|
||||
ON CONFLICT DO NOTHING""",
|
||||
today, portfolio_value, daily_pnl, daily_orders or 0,
|
||||
)
|
||||
logger.info("Captured risk snapshot: pnl=%.2f trades=%d", daily_pnl, daily_orders or 0)
|
||||
except Exception:
|
||||
logger.exception("Failed to capture risk snapshot")
|
||||
```
|
||||
|
||||
**Validates:** Bugfix 2.6, 2.7; Regression 3.10
|
||||
|
||||
---
|
||||
|
||||
### Fix 5: Prediction Price Fallback (Bug 1.8)
|
||||
|
||||
**File:** `services/validation/prediction_snapshot.py`
|
||||
|
||||
After the primary `market_snapshots` price lookup returns NULL, add a fallback:
|
||||
|
||||
```python
|
||||
# After market_snapshots lookup:
|
||||
if price_at_prediction is None:
|
||||
pos_row = await conn.fetchrow(
|
||||
"SELECT current_price FROM positions "
|
||||
"WHERE ticker = $1 AND current_price IS NOT NULL LIMIT 1",
|
||||
ticker,
|
||||
)
|
||||
if pos_row:
|
||||
price_at_prediction = float(pos_row["current_price"])
|
||||
```
|
||||
|
||||
Only covers tickers with open positions (currently 10). Acceptable tradeoff — most active tickers are the ones we hold.
|
||||
|
||||
**Validates:** Bugfix 2.8; Regression 3.4
|
||||
|
||||
---
|
||||
|
||||
### Fix 6: Lake Publisher Scale-Down (Bug 1.9)
|
||||
|
||||
**File:** `infra/helm/stonks-oracle/values.yaml`
|
||||
|
||||
```yaml
|
||||
# Change:
|
||||
replicas: 0
|
||||
```
|
||||
|
||||
Keeps the deployment definition intact for future use but schedules no pods.
|
||||
|
||||
**Validates:** Bugfix 2.9; Regression 3.5
|
||||
|
||||
---
|
||||
|
||||
### Fix 7: Order rejection_reason Population (Bug 1.10)
|
||||
|
||||
**File:** `services/adapters/broker_service.py`
|
||||
|
||||
Extend `_INSERT_ORDER` to include `rejection_reason` and `rejected_at`:
|
||||
|
||||
```python
|
||||
_INSERT_ORDER = """
|
||||
INSERT INTO orders (
|
||||
id, recommendation_id, broker_account_id, ticker, side, order_type,
|
||||
quantity, limit_price, stop_price, status, idempotency_key,
|
||||
broker_order_id, decision_trace, submitted_at, filled_at,
|
||||
fill_price, fill_quantity, rejection_reason, rejected_at
|
||||
) VALUES (
|
||||
$1::uuid, $2, $3::uuid, $4, $5, $6,
|
||||
$7, $8, $9, $10, $11,
|
||||
$12, $13::jsonb, $14, $15,
|
||||
$16, $17, $18, $19
|
||||
)
|
||||
ON CONFLICT (idempotency_key) DO UPDATE SET
|
||||
status = EXCLUDED.status,
|
||||
broker_order_id = EXCLUDED.broker_order_id,
|
||||
filled_at = EXCLUDED.filled_at,
|
||||
fill_price = EXCLUDED.fill_price,
|
||||
fill_quantity = EXCLUDED.fill_quantity,
|
||||
rejection_reason = COALESCE(EXCLUDED.rejection_reason, orders.rejection_reason),
|
||||
rejected_at = COALESCE(EXCLUDED.rejected_at, orders.rejected_at),
|
||||
updated_at = NOW()
|
||||
"""
|
||||
```
|
||||
|
||||
Update `persist_order()` to pass the new parameters:
|
||||
|
||||
```python
|
||||
rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None
|
||||
rejected_at = now if resp.status == OrderStatus.REJECTED else None
|
||||
# Add as params $18, $19
|
||||
```
|
||||
|
||||
**Validates:** Bugfix 2.10; Regression 3.6, 3.8
|
||||
|
||||
---
|
||||
|
||||
## Correctness Properties
|
||||
|
||||
Property 1: Rate limit compliance — After fix, the rolling 1-minute window for Polygon requests SHALL NOT exceed the configured limit (default 5). Existing `check_rate_limit()` windowed counter enforces this; we only change the threshold constant.
|
||||
|
||||
Property 2: Validation cycle completeness — `prediction_outcomes` row count SHALL grow monotonically after the first validation cycle runs. Each run finds matured snapshots not yet evaluated and persists outcomes.
|
||||
|
||||
Property 3: Metric snapshot freshness — `model_metric_snapshots` SHALL contain rows with `generated_at` within the last 2 hours after 2+ validation cycles. The quality gate can then evaluate against real data.
|
||||
|
||||
Property 4: Config read correctness — `_read_v3_flag()` SHALL return True when `risk_configs.config->>'v3_engine_enabled'` is `'true'` and False for all other values including NULL or missing key.
|
||||
|
||||
Property 5: Snapshot idempotency — `portfolio_snapshots` SHALL contain at most 1 row per `snapshot_date`. The `maybe_capture_daily_snapshots` function checks for existing rows before insert.
|
||||
|
||||
Property 6: Rejection reason preservation — Every order with `status = 'rejected'` persisted via `persist_order()` SHALL have a non-NULL `rejection_reason` extracted from the error response.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests:** Update `test_scheduler.py` with a test verifying `run_validation_cycle` is called after the counter threshold. Test `_read_v3_flag` with mocked JSONB config returning various values.
|
||||
- **Integration tests:** Verify `persist_order` with rejected status populates `rejection_reason` column.
|
||||
- **Manual verification post-deploy:**
|
||||
- `kubectl logs deployment/scheduler -n stonks-beta --tail=100 | grep Validation` shows outcome counts
|
||||
- `SELECT count(*) FROM prediction_outcomes` starts growing within 1 hour
|
||||
- `SELECT count(*) FROM model_metric_snapshots` populates after outcomes exist
|
||||
- Scheduler logs show significantly fewer "Rate limit hit" warnings
|
||||
- Aggregation logs no longer show "column value does not exist" error
|
||||
- After market close: `SELECT * FROM portfolio_snapshots WHERE snapshot_date = CURRENT_DATE` returns 1 row
|
||||
|
||||
## Glossary
|
||||
|
||||
| Term | Definition |
|
||||
|------|-----------|
|
||||
| Validation cycle | Hourly scheduler task: evaluate_matured_predictions → compute_and_store_metric_snapshots |
|
||||
| Quality gate | Threshold check on model_metric_snapshots that determines if trading can be promoted from paper to live |
|
||||
| Prediction snapshot | Frozen state of a recommendation at generation time (prices, evidence, scores) |
|
||||
| Outcome evaluation | Matching a matured prediction snapshot against realized market returns |
|
||||
| Polygon free tier | API plan with ~5 requests/minute rate limit |
|
||||
@@ -0,0 +1,69 @@
|
||||
# Implementation Plan: ops-pipeline-fixes
|
||||
|
||||
## Overview
|
||||
|
||||
Fix 10 operational bugs preventing the validation/calibration feedback loop from functioning and degrading ingestion throughput. Changes span scheduler (rate limiting + periodic tasks), aggregation worker (config query), broker service (rejection reason), prediction snapshot (price fallback), and Helm chart (dead pod removal).
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] 1. Fix Polygon global rate limit — In `services/scheduler/app.py`, replace `POLYGON_GLOBAL_RATE_LIMIT: int = 45` with `POLYGON_GLOBAL_RATE_LIMIT: int = int(os.getenv("POLYGON_GLOBAL_RATE_LIMIT", "5"))` to make it env-configurable and default to the free-tier limit
|
||||
- **Validates: Bugfix 2.1; Regression 3.1, 3.7**
|
||||
|
||||
- [x] 2. Fix v3_engine_enabled query — In `services/aggregation/worker.py`, replace `_V3_ENGINE_FLAG_QUERY` from `SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'` to `SELECT config->>'v3_engine_enabled' AS enabled FROM risk_configs WHERE name = 'default' AND active = TRUE LIMIT 1`, and rewrite `_read_v3_flag()` to parse the returned string (checking for "true"/"1"/"yes"), returning False for NULL/missing/error
|
||||
- **Validates: Bugfix 2.2; Regression 3.2**
|
||||
|
||||
- [x] 3. Add validation cycle constant and counter — In `services/scheduler/app.py`, add `VALIDATION_CYCLE_INTERVAL = int(os.getenv("VALIDATION_CYCLE_INTERVAL", "240"))` constant and `validation_counter = 0` initialization in `main()`
|
||||
- **Validates: Bugfix 2.3, 2.4**
|
||||
|
||||
- [x] 4. Implement run_validation_cycle function — In `services/scheduler/app.py`, implement `run_validation_cycle(pool)` that calls `evaluate_matured_predictions(pool)` followed by `compute_and_store_metric_snapshots(pool)`, with try/except logging for each and skipping metrics if outcomes fail
|
||||
- **Validates: Bugfix 2.3, 2.4, 2.5; Regression 3.3**
|
||||
|
||||
- [x] 5. Wire validation cycle into main loop — In `services/scheduler/app.py` main loop, add the counter increment and conditional call to `run_validation_cycle(pool)` after the existing `report_schedule_counter` block
|
||||
- **Validates: Bugfix 2.3, 2.4, 2.5**
|
||||
|
||||
- [x] 6. Add snapshot cycle constant and counter — In `services/scheduler/app.py`, add `SNAPSHOT_CYCLE_INTERVAL = int(os.getenv("SNAPSHOT_CYCLE_INTERVAL", "240"))` constant and `snapshot_counter = 0` initialization in `main()`
|
||||
- **Validates: Bugfix 2.6, 2.7**
|
||||
|
||||
- [x] 7. Implement maybe_capture_daily_snapshots function — In `services/scheduler/app.py`, implement `maybe_capture_daily_snapshots(pool)` that checks time (after 16:30 ET), checks idempotency (no existing row for today), queries positions table for portfolio value/unrealized PnL, and inserts into `portfolio_snapshots` and `daily_risk_snapshots`
|
||||
- **Validates: Bugfix 2.6, 2.7; Regression 3.10**
|
||||
|
||||
- [x] 8. Wire snapshot cycle into main loop — In `services/scheduler/app.py` main loop, add the counter increment and conditional call to `maybe_capture_daily_snapshots(pool)` after the validation counter block
|
||||
- **Validates: Bugfix 2.6, 2.7**
|
||||
|
||||
- [x] 9. Add prediction price fallback — In `services/validation/prediction_snapshot.py`, after the primary market_snapshots price lookup returns NULL for `price_at_prediction`, add a fallback query to positions table: `SELECT current_price FROM positions WHERE ticker = $1 AND current_price IS NOT NULL LIMIT 1`
|
||||
- **Validates: Bugfix 2.8; Regression 3.4**
|
||||
|
||||
- [x] 10. Scale down lake-publisher — In `infra/helm/stonks-oracle/values.yaml`, change the lake-publisher `replicas` from `1` to `0`
|
||||
- **Validates: Bugfix 2.9; Regression 3.5**
|
||||
|
||||
- [x] 11. Extend _INSERT_ORDER SQL — In `services/adapters/broker_service.py`, extend `_INSERT_ORDER` SQL to include `rejection_reason` and `rejected_at` as parameters $18 and $19, with COALESCE in the ON CONFLICT UPDATE clause to preserve existing values
|
||||
- **Validates: Bugfix 2.10; Regression 3.6, 3.8**
|
||||
|
||||
- [x] 12. Update persist_order parameters — In `services/adapters/broker_service.py`, update `persist_order()` to compute `rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None` and `rejected_at = now if resp.status == OrderStatus.REJECTED else None`, passing them as the final two parameters in the execute call
|
||||
- **Validates: Bugfix 2.10; Regression 3.6, 3.8**
|
||||
|
||||
- [x] 13. Lint and test — Run `.venv/bin/ruff check services/` and `.venv/bin/python -m pytest tests/ -x --tb=short -q` to verify no regressions
|
||||
- **Validates: Regression 3.1–3.10**
|
||||
|
||||
## Task Dependency Graph
|
||||
|
||||
```json
|
||||
{
|
||||
"waves": [
|
||||
{"tasks": [1, 2, 9, 10]},
|
||||
{"tasks": [3, 6, 11]},
|
||||
{"tasks": [4, 7, 12]},
|
||||
{"tasks": [5, 8]},
|
||||
{"tasks": [13]}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Tasks 1, 2, 9, 10 are fully independent. Tasks 3/6/11 set up constants needed by 4/7/12. Tasks 5/8 wire into the main loop after their functions exist. Task 13 validates everything last.
|
||||
|
||||
## Notes
|
||||
|
||||
- No database migrations required — all tables already exist with correct columns
|
||||
- All scheduler changes use the existing counter-based periodic task pattern already established for cleanup, aggregation, and report tasks
|
||||
- Lazy imports in `run_validation_cycle` avoid circular imports and keep scheduler startup fast
|
||||
- The `maybe_capture_daily_snapshots` idempotency check prevents duplicate rows on scheduler restart
|
||||
Binary file not shown.
Reference in New Issue
Block a user