feat: Intelligence Pipeline v3 — full implementation
Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.
This commit is contained in:
Celes Renata
@@ -0,0 +1,363 @@
|
||||
# V3 Annotation Guidelines
|
||||
|
||||
**Schema version:** 1.0.0
|
||||
**Last updated:** 2025-01-15
|
||||
|
||||
## Purpose
|
||||
|
||||
These guidelines define how human annotators and automated systems label documents in the Intelligence Pipeline v3 Gold Corpus. Every annotation must be evidence-grounded — no label is valid without a supporting evidence span traceable to the source text.
|
||||
|
||||
## Core Principles
|
||||
|
||||
1. **Evidence first.** If you cannot point to exact text that supports a label, do not apply the label.
|
||||
2. **Explicit over inferred.** Mark only what the document explicitly states in primary annotations. Inferred exposure uses a separate, lower-confidence channel.
|
||||
3. **Precision over recall.** A missed entity is preferable to a fabricated one. The pipeline uses multiple stages — later stages catch omissions.
|
||||
4. **Reproducibility.** Two annotators given the same document should produce substantially the same labels. Ambiguous cases are marked, not resolved by guess.
|
||||
|
||||
---
|
||||
|
||||
## Evidence Spans
|
||||
|
||||
### Definition
|
||||
|
||||
An evidence span is the exact substring of the source document that supports an annotation. It uses zero-based character offsets into the original (pre-chunking) document text.
|
||||
|
||||
### Rules
|
||||
|
||||
- Every entity, event, relation, numeric fact, and sentiment annotation MUST reference at least one evidence span.
|
||||
- Spans should be minimal but complete — include enough context for the label to be verifiable without the full document.
|
||||
- Overlapping spans are permitted (e.g., the same sentence supports both an entity and an event).
|
||||
- The `text` field MUST exactly match `source_text[start_char:end_char]`.
|
||||
|
||||
### Positive example
|
||||
|
||||
```
|
||||
Source: "Apple Inc. reported quarterly earnings of $1.52 per share"
|
||||
Span: start_char=0, end_char=10, text="Apple Inc."
|
||||
```
|
||||
|
||||
### Negative example
|
||||
|
||||
```
|
||||
Source: "Apple Inc. reported quarterly earnings of $1.52 per share"
|
||||
Span: start_char=0, end_char=5, text="Apple"
|
||||
```
|
||||
❌ Truncating "Apple Inc." to "Apple" loses the corporate suffix needed to distinguish from Apple Records or the fruit.
|
||||
|
||||
---
|
||||
|
||||
## Entity Annotation
|
||||
|
||||
### Entity Types
|
||||
|
||||
| Type | When to use | Example |
|
||||
|------|-------------|---------|
|
||||
| `company` | Legal entity, publicly traded firm, government agency | "Apple Inc.", "The Federal Reserve" |
|
||||
| `person` | Named individual | "Tim Cook", "Jerome Powell" |
|
||||
| `product` | Named product or service | "iPhone 16", "Azure OpenAI Service" |
|
||||
| `event` | Named event instance | "Q1 2025 earnings call" |
|
||||
| `financial_metric` | Named metric class | "EPS", "revenue", "free cash flow" |
|
||||
| `date` | Temporal expression | "Q1 2025", "January 15, 2025" |
|
||||
| `percentage` | Percentage value | "4%", "25 basis points" |
|
||||
| `currency` | Monetary value | "$1.52", "$10 billion" |
|
||||
| `relationship` | Explicit relationship mention | "subsidiary", "joint venture partner" |
|
||||
|
||||
### Canonical Resolution
|
||||
|
||||
- If an entity maps to a company in the symbol registry, set `canonical_id` and `canonical_name` (ticker).
|
||||
- If an entity is ambiguous (e.g., "Apple" could be AAPL or a fruit company), mark an ambiguity marker and set confidence below 1.0.
|
||||
- Do NOT invent canonical IDs. If not in the registry, leave `canonical_id` as null.
|
||||
|
||||
### Positive example
|
||||
|
||||
```json
|
||||
{
|
||||
"entity_type": "company",
|
||||
"literal_text": "Alphabet",
|
||||
"canonical_id": "googl-uuid",
|
||||
"canonical_name": "GOOGL",
|
||||
"confidence": 0.97
|
||||
}
|
||||
```
|
||||
|
||||
### Negative example
|
||||
|
||||
```json
|
||||
{
|
||||
"entity_type": "company",
|
||||
"literal_text": "the company",
|
||||
"canonical_id": "aapl-uuid",
|
||||
"canonical_name": "AAPL",
|
||||
"confidence": 0.90
|
||||
}
|
||||
```
|
||||
❌ "the company" is a pronoun reference, not an entity mention. Resolve coreference but annotate the actual named mention, not the pronoun.
|
||||
|
||||
---
|
||||
|
||||
## Event Classification
|
||||
|
||||
### Event Classes
|
||||
|
||||
| Class | Definition | Distinguishing criteria |
|
||||
|-------|-----------|------------------------|
|
||||
| `earnings_beat` | Reported EPS or revenue exceeds consensus | Explicit comparison to estimates |
|
||||
| `earnings_miss` | Reported EPS or revenue below consensus | Explicit comparison to estimates |
|
||||
| `guidance_raise` | Forward guidance raised vs prior or consensus | Future-looking, not historical result |
|
||||
| `guidance_cut` | Forward guidance lowered | Future-looking, not historical result |
|
||||
| `ma_announcement` | Merger, acquisition, investment, or divestiture | Transaction between entities |
|
||||
| `legal_regulatory` | Lawsuit, fine, regulatory action, or settlement | Legal or regulatory body involved |
|
||||
| `product_launch` | New product, service, or major feature announced | Not routine updates |
|
||||
| `supply_chain` | Disruption, partnership, or change in supply relationships | Affects production/delivery |
|
||||
| `rating_change` | Analyst upgrade, downgrade, or target change | From research analyst/firm |
|
||||
| `management_change` | CEO/CFO/board appointment, resignation, or removal | C-suite or board level |
|
||||
| `macro_event` | Interest rates, policy, trade, geopolitical | Not specific to one company |
|
||||
| `dividend_change` | Dividend increase, decrease, or special dividend | Shareholder distribution |
|
||||
| `buyback` | Share repurchase program announcement or completion | Capital return via buyback |
|
||||
|
||||
### Adjudication triggers for events
|
||||
|
||||
Route to the 9B adjudicator when:
|
||||
- The same facts could be classified as multiple event types (e.g., guidance_raise during an earnings call could be either earnings_beat or guidance_raise — label the most specific applicable class).
|
||||
- The event is implied but not explicitly stated.
|
||||
- The primary company is unclear.
|
||||
|
||||
### Positive example
|
||||
|
||||
```
|
||||
Source: "Apple beat earnings expectations with EPS of $1.52 vs $1.43 expected"
|
||||
Event class: earnings_beat
|
||||
Confidence: 0.98
|
||||
```
|
||||
|
||||
### Negative example
|
||||
|
||||
```
|
||||
Source: "Apple reported EPS of $1.52"
|
||||
Event class: earnings_beat
|
||||
```
|
||||
❌ Without a comparison to consensus/estimates, this is a numeric fact report, not an earnings beat. The document must provide evidence of beating expectations.
|
||||
|
||||
---
|
||||
|
||||
## Relations
|
||||
|
||||
### Relation Types
|
||||
|
||||
| Type | Subject | Object | When to use |
|
||||
|------|---------|--------|-------------|
|
||||
| `directly_affects` | Event | Company | Event explicitly names or discusses the company |
|
||||
| `inferred_exposure` | Event | Company | Exposure inferred from sector, supply chain, or competition |
|
||||
| `competes_with` | Company | Company | Competitive relationship stated or clearly implied |
|
||||
| `supplies` | Company | Company | Supply chain relationship stated |
|
||||
|
||||
### Critical distinction: directly_affects vs inferred_exposure
|
||||
|
||||
- `directly_affects`: The document **explicitly states** the company is impacted. Evidence span exists.
|
||||
- `inferred_exposure`: The impact is **reasoned** from relationships, not stated. May have weak or no direct evidence span.
|
||||
|
||||
Only `directly_affects` enters primary company extraction. `inferred_exposure` flows through the separate interpolation/propagation architecture with distinct confidence and provenance.
|
||||
|
||||
### Positive example (directly_affects)
|
||||
|
||||
```
|
||||
Source: "Microsoft announced a $10 billion investment in OpenAI"
|
||||
Relation: directly_affects(event=ma_announcement, company=Microsoft)
|
||||
Evidence: "Microsoft announced"
|
||||
```
|
||||
|
||||
### Negative example (incorrectly using directly_affects)
|
||||
|
||||
```
|
||||
Source: "Microsoft announced a $10 billion investment in OpenAI"
|
||||
Relation: directly_affects(event=ma_announcement, company=Google)
|
||||
```
|
||||
❌ Google is not mentioned in the event sentence. This should be `inferred_exposure` based on competitive relationship, with appropriate lower confidence.
|
||||
|
||||
---
|
||||
|
||||
## Numeric Facts
|
||||
|
||||
### Annotation rules
|
||||
|
||||
1. Always store both `literal_value` (exact text) and `normalized_value` (parsed number).
|
||||
2. Include `unit` (USD, %, bps, shares, etc.).
|
||||
3. Link to the subject entity when determinable.
|
||||
4. Use `predicate` to capture the semantic role: reported, expected, raised_to, cut_to, beat_by, missed_by.
|
||||
5. Include `period` when the fact references a specific time frame.
|
||||
|
||||
### Normalization conventions
|
||||
|
||||
| Literal | Normalized | Unit |
|
||||
|---------|-----------|------|
|
||||
| "$1.52" | 1.52 | USD |
|
||||
| "$94.9 billion" | 94900000000 | USD |
|
||||
| "25 basis points" | 0.25 | percentage_points |
|
||||
| "4%" | 4.0 | % |
|
||||
| "$0.26 per share" | 0.26 | USD |
|
||||
|
||||
### Positive example
|
||||
|
||||
```json
|
||||
{
|
||||
"fact_type": "eps",
|
||||
"predicate": "reported",
|
||||
"literal_value": "$1.52 per share",
|
||||
"normalized_value": 1.52,
|
||||
"unit": "USD",
|
||||
"period": {"period_type": "fiscal_quarter", "fiscal_year": 2025, "fiscal_quarter": 1}
|
||||
}
|
||||
```
|
||||
|
||||
### Negative example
|
||||
|
||||
```json
|
||||
{
|
||||
"fact_type": "eps",
|
||||
"predicate": "reported",
|
||||
"literal_value": "$1.52 per share",
|
||||
"normalized_value": 152,
|
||||
"unit": "cents"
|
||||
}
|
||||
```
|
||||
❌ While $1.52 = 152 cents, always normalize to the unit stated in the source. Conversion to a different unit introduces potential confusion.
|
||||
|
||||
---
|
||||
|
||||
## Sentiment
|
||||
|
||||
### Rules
|
||||
|
||||
1. Sentiment is **company-specific**, not document-level. A single article can have positive sentiment for one company and negative for another.
|
||||
2. Annotate probability distributions (positive, negative, neutral) that sum to 1.0.
|
||||
3. `mixed` label is used when evidence groups disagree — it is computed from evidence-group-level disagreement, NOT an unconstrained fourth class.
|
||||
4. The label should reflect the dominant probability.
|
||||
|
||||
### When to label "mixed"
|
||||
|
||||
Label `mixed` when:
|
||||
- Different paragraphs contain opposing sentiment for the same company
|
||||
- The same fact has both positive and negative implications (e.g., restructuring = cost cuts but also layoffs)
|
||||
- Analyst opinions explicitly disagree within the document
|
||||
|
||||
Do NOT label `mixed` when:
|
||||
- Sentiment is merely uncertain or mild — that's `neutral` with lower confidence
|
||||
- The document discusses multiple companies with different sentiments — annotate separately per company
|
||||
|
||||
### Positive example
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "mixed",
|
||||
"positive_probability": 0.40,
|
||||
"negative_probability": 0.45,
|
||||
"neutral_probability": 0.15,
|
||||
"evidence_ids": ["ev-pressure", "ev-validation"]
|
||||
}
|
||||
```
|
||||
(Article says AI investment pressures cloud revenue but validates the broader thesis)
|
||||
|
||||
### Negative example
|
||||
|
||||
```json
|
||||
{
|
||||
"label": "mixed",
|
||||
"positive_probability": 0.85,
|
||||
"negative_probability": 0.05,
|
||||
"neutral_probability": 0.10
|
||||
}
|
||||
```
|
||||
❌ When positive_probability dominates at 0.85, the label should be `positive`, not `mixed`. Mixed requires genuine disagreement in evidence.
|
||||
|
||||
---
|
||||
|
||||
## Direct Effects vs Inferred Exposure
|
||||
|
||||
### Direct Effects
|
||||
|
||||
A direct effect means the document **explicitly states or clearly demonstrates** that an event impacts a specific company.
|
||||
|
||||
**Criteria:**
|
||||
- The company is named in the same sentence or paragraph as the event
|
||||
- The causal link is stated, not inferred
|
||||
- Evidence span directly connects event to company
|
||||
|
||||
### Inferred Exposure
|
||||
|
||||
Inferred exposure captures **reasoned but unstated** impacts on companies.
|
||||
|
||||
**Criteria:**
|
||||
- The company is NOT explicitly linked to the event in the source text
|
||||
- The connection comes from known relationships (competitor, supplier, sector peer)
|
||||
- Confidence should be lower than direct effects (typically 0.5–0.8)
|
||||
- Requires `reasoning` field explaining the inference chain
|
||||
|
||||
### Adjudication routing
|
||||
|
||||
When it's unclear whether an effect is direct or inferred, mark an ambiguity marker with type `implied_causal_impact` and route to the 9B adjudicator.
|
||||
|
||||
---
|
||||
|
||||
## Ambiguity Markers
|
||||
|
||||
### When to flag
|
||||
|
||||
Flag ambiguity when:
|
||||
- An alias resolves to multiple candidate companies (`unresolved_alias`)
|
||||
- Multiple companies could be the primary subject (`multiple_primary_companies`)
|
||||
- Numeric facts within the same document contradict each other (`contradictory_numeric_facts`)
|
||||
- Sentiment evidence points in opposing directions for the same company (`conflicting_sentiment`)
|
||||
- Impact is implied through causal chain, not stated (`implied_causal_impact`)
|
||||
- Guidance must be compared to consensus to determine direction (`guidance_vs_consensus_requires_reasoning`)
|
||||
- A required field cannot be determined from available evidence (`material_field_missing`)
|
||||
- Evidence covers less than the minimum threshold for confident extraction (`evidence_coverage_below_threshold`)
|
||||
- Calibrated confidence falls below the routing threshold (`calibrated_confidence_below_threshold`)
|
||||
- A relation spans multiple document chunks (`long_document_cross_chunk_relation`)
|
||||
|
||||
### Severity levels
|
||||
|
||||
- **low**: The annotation is likely correct but has reduced certainty. Fast path may proceed with a confidence penalty.
|
||||
- **medium**: The annotation requires review. Routes to adjudication by default.
|
||||
- **high**: The annotation cannot be reliably made without semantic reasoning. Always routes to adjudication.
|
||||
|
||||
---
|
||||
|
||||
## Safety-Critical Fields
|
||||
|
||||
The following fields are **safety-critical** for promotion gates. Errors in these fields can directly cause incorrect trading decisions:
|
||||
|
||||
| Field | Why it's critical | Minimum promotion gate |
|
||||
|-------|-------------------|----------------------|
|
||||
| Company identity (ticker) | Wrong ticker = trade on wrong security | Precision ≥ 0.95, Recall ≥ 0.90 |
|
||||
| Event class | Misclassifying beat/miss inverts signal direction | Macro-F1 ≥ 0.85 |
|
||||
| Sentiment direction | Wrong sentiment → wrong position direction | Direction accuracy ≥ 0.90 |
|
||||
| Numeric fact values | Wrong magnitude affects impact estimation | Tolerance match ≥ 0.92 |
|
||||
| Direct effect attribution | Wrong company attribution creates false signals | Precision ≥ 0.93 |
|
||||
| Evidence support | Unsupported claims are unverifiable | Support rate ≥ 0.95 |
|
||||
| Confidence calibration | Overconfidence bypasses review | ECE ≤ 0.05 |
|
||||
|
||||
Annotators must pay special attention to these fields. During review, any error in a safety-critical field requires correction before the annotation can receive "gold" status.
|
||||
|
||||
---
|
||||
|
||||
## Annotation Workflow
|
||||
|
||||
1. **First pass:** Identify all entities and evidence spans
|
||||
2. **Second pass:** Classify events and link to companies
|
||||
3. **Third pass:** Extract numeric facts with periods
|
||||
4. **Fourth pass:** Assess per-company sentiment
|
||||
5. **Fifth pass:** Identify relations, direct effects, and inferred exposures
|
||||
6. **Sixth pass:** Flag ambiguities and set confidence levels
|
||||
7. **Review:** Senior annotator validates safety-critical fields
|
||||
|
||||
### Inter-annotator agreement
|
||||
|
||||
Hard cases (flagged with ambiguity markers) receive double annotation. Inter-annotator agreement is measured per field type using Cohen's kappa. Target: κ ≥ 0.80 for entity and event labels, κ ≥ 0.70 for relations and sentiment.
|
||||
|
||||
---
|
||||
|
||||
## Version History
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------|---------|
|
||||
| 1.0.0 | 2025-01-15 | Initial schema and guidelines |
|
||||
@@ -0,0 +1,84 @@
|
||||
# Session Context — July 11, 2026
|
||||
|
||||
## Current State
|
||||
|
||||
### Active Namespace: `stonks-beta`
|
||||
- This is the ONLY namespace that should be running
|
||||
- `stonks-oracle` namespace has been scaled to 0 replicas (all deployments)
|
||||
- Dashboard: `https://stonks-beta.celestium.life`
|
||||
- API: `https://stonks-api-beta.celestium.life`
|
||||
|
||||
### What Was Done This Session
|
||||
|
||||
#### 1. Pipeline Health Fixes (spec: `.kiro/specs/pipeline-health-fixes/`)
|
||||
All implemented and deployed:
|
||||
- **Stuck Parsed Docs**: `STALE_PARSED_THRESHOLD_MINUTES` 240→30, `LIMIT` 100→500, `_ENQUEUED_TTL` 14400→3600 (`services/scheduler/app.py`)
|
||||
- **Price Fallback**: Added 24h market_snapshots time-window fallback in `create_prediction_snapshot()` (`services/validation/prediction_snapshot.py`)
|
||||
- **Sentiment Normalization**: Added `normalize_impact_scores()` z-score function (`services/aggregation/scoring.py`) + integrated into `aggregate_company_window()` (`services/aggregation/worker.py`)
|
||||
- **Signal Engine**: Replicas set to 0 in all Helm values files
|
||||
- **Quality Gate**: `max_snapshot_age_hours` 24→48 (`services/trading/model_quality_gate.py`)
|
||||
- **Backfill script**: `scripts/backfill_snapshot_prices.py` (one-time, not yet run on beta)
|
||||
|
||||
#### 2. Extractor Null-Field Fix
|
||||
- `services/extractor/schemas.py`: `_normalize_extraction_data()` now handles `None` values (not just missing keys) and filters out company entries with empty ticker
|
||||
- Test updated: `tests/test_extractor_schemas.py::test_validate_semantic_missing_ticker_is_error`
|
||||
|
||||
#### 3. Macro Doc Status Fix
|
||||
- `services/extractor/main.py`: `_process_macro_classification()` now updates document status to 'extracted' on success, 'extraction_failed' on error
|
||||
- Beta DB: manually fixed 1849 stuck macro docs (UPDATE status='extracted' WHERE id IN global_events)
|
||||
|
||||
#### 4. Dashboard Fix
|
||||
- `frontend/src/pages/OpsPipeline.tsx`: Document Stages now uses time-filtered `/health` data (consistent with other sections), all-time from SSE stream shown as subtitle, time range labels added to all sections
|
||||
|
||||
#### 5. CI/CD DNS Fix
|
||||
- `.woodpecker/*.yml`: All 5 pipeline files now use `clone.git.settings.remote: http://10.43.73.77:3000/admin/stonks-oracle.git` (Gitea ClusterIP directly, bypasses DNS)
|
||||
- CoreDNS: scaled to 4 replicas, `forward . 192.168.42.1`, `dnsPolicy: None` with `nameservers: [192.168.42.1]`
|
||||
- Woodpecker: `WOODPECKER_BACKEND_K8S_DNS_CONFIG` has `nameservers:[10.43.0.10]` + searches including `git-server.svc.cluster.local`
|
||||
|
||||
### Known Issues / TODO
|
||||
|
||||
1. **`stonks-oracle` namespace**: Scaled to 0 but still exists with stale data (42K extraction queue in Redis DB 0). Could be cleaned up or deleted entirely.
|
||||
|
||||
2. **Thesis Rewriter agent**: Was hammering vLLM from stonks-oracle namespace (5600+ calls/24h). Now stopped since namespace is scaled down. If it was also running in beta, check if recommendation service is calling vLLM for thesis rewrites excessively.
|
||||
|
||||
3. **`AxionML/Qwen3.5-9B-NVFP4` requests**: Something external is hitting vLLM with a model that doesn't exist (404s). Not from our pipeline — likely Open WebUI or another tool on the network configured with wrong model name. Source IP: goes through `vllm-metrics` nginx proxy (`10.42.1.155`).
|
||||
|
||||
4. **GitHub mirror**: `finalize.yml` mirror-github step fails (SSH key or DNS). Has `failure: ignore` so non-blocking. Needs `github_ssh_key` secret configured in Woodpecker.
|
||||
|
||||
5. **OpsPipeline dashboard**: Numbers now show time-filtered data. The "Document Stages" section shows counts from the selected time window (default 24h), with all-time totals as subtle subtitles. Currently beta shows: extracted=5417, low_quality=1659, parsed=15.
|
||||
|
||||
6. **Aggregation not generating trends on weekends**: Expected — market hours check prevents weekend trend generation. Will resume Monday.
|
||||
|
||||
7. **15 docs still in `parsed` status**: These are likely fresh ingests waiting for the next extraction cycle. Not stuck.
|
||||
|
||||
### Agent Performance (beta, last 24h as of session end)
|
||||
- Document Intelligence Extractor: 33 calls, 94% success, avg 11.4s, conf 0.794
|
||||
- Global Event Classifier: 81 calls, 99% success, avg 4.2s, conf 0.745
|
||||
- Thesis Rewriter: 5603 calls, 100% success, avg 2.5s (from stonks-oracle before shutdown)
|
||||
- Report Summarizer: 6 calls, 100% success, avg 6.9s
|
||||
|
||||
### Infrastructure
|
||||
- k3s cluster: 4 NixOS nodes (gremlin-1 through gremlin-4)
|
||||
- vLLM: `vllm-service` namespace, model `numind/NuExtract3`, 4070 Ti Super 16GB
|
||||
- CoreDNS: 4 replicas, `forward . 192.168.42.1`
|
||||
- Redis: DB 0 = stonks-oracle (stale), DB 1 = stonks-beta (active)
|
||||
- PostgreSQL: shared instance, both namespaces use same DB server (different databases? or same? — needs verification)
|
||||
- Gitea: `git-server` namespace, ClusterIP 10.43.73.77:3000, NodePort 30300
|
||||
- Woodpecker: `woodpecker` namespace, kubernetes backend, 2 agents
|
||||
|
||||
### Key Files Modified
|
||||
```
|
||||
services/scheduler/app.py — recovery thresholds + batch limit
|
||||
services/validation/prediction_snapshot.py — 24h price fallback
|
||||
services/aggregation/scoring.py — normalize_impact_scores()
|
||||
services/aggregation/worker.py — normalization integration
|
||||
services/trading/model_quality_gate.py — 48h threshold
|
||||
services/extractor/schemas.py — null field handling
|
||||
services/extractor/main.py — macro doc status update
|
||||
frontend/src/pages/OpsPipeline.tsx — dashboard fix
|
||||
scripts/backfill_snapshot_prices.py — new script
|
||||
tests/test_pbt_pipeline_health_*.py — PBT tests
|
||||
tests/test_extractor_schemas.py — updated test
|
||||
infra/helm/stonks-oracle/values*.yaml — signal-engine replicas
|
||||
.woodpecker/*.yml — ClusterIP clone fix
|
||||
```
|
||||
@@ -0,0 +1,144 @@
|
||||
# Stonks Oracle — What It Is and What It Does
|
||||
|
||||
## The One-Liner
|
||||
|
||||
Stonks Oracle is an autonomous market intelligence system that reads the news so you don't have to, forms a view on 50 publicly traded companies, and paper-trades that view — then grades its own homework.
|
||||
|
||||
---
|
||||
|
||||
## The Problem It Solves
|
||||
|
||||
Markets are noisy. Every day, hundreds of news articles, SEC filings, earnings transcripts, and geopolitical headlines hit the wire. A human analyst covering even a dozen names struggles to weigh all of it in real time. Most retail and even some institutional desks end up reacting to headlines rather than synthesizing the full picture.
|
||||
|
||||
Stonks Oracle replaces that manual synthesis with an always-on pipeline:
|
||||
|
||||
1. **It reads everything.** News articles, 10-K/10-Q filings, earnings calls, press releases, and macro/geopolitical headlines — ingested automatically on a schedule.
|
||||
2. **It extracts structured intelligence.** A local AI model reads each document and pulls out: which companies are mentioned, the sentiment (bullish / bearish / neutral), the catalyst type (earnings, product launch, regulatory action, M&A, etc.), impact horizon (same-day through 90 days), key facts, and material risks.
|
||||
3. **It forms a view.** Those individual extractions are aggregated into rolling trend summaries per company, refreshed continuously. The system flags contradictions (e.g., one filing is bullish but a news article is bearish) and tracks confidence based on evidence depth.
|
||||
4. **It decides whether to trade.** When confidence is high enough, contradiction is low, and evidence is fresh, it issues a buy or sell recommendation — with a full written thesis explaining why.
|
||||
5. **It executes paper trades.** An autonomous trading engine places orders through Alpaca's paper-trading system. Position sizing, stop-losses, take-profits, sector concentration limits, and circuit breakers are all built in.
|
||||
6. **It measures itself.** Every prediction is frozen at the moment it's made, then checked against actual price movements days and weeks later. The system tracks its own win rate, calibration, and whether it's beating SPY.
|
||||
|
||||
---
|
||||
|
||||
## The Universe
|
||||
|
||||
50 companies across 10 sectors:
|
||||
|
||||
| Sector | Examples |
|
||||
|--------|----------|
|
||||
| Technology | AAPL, MSFT, NVDA, GOOGL, META |
|
||||
| Consumer Cyclical | AMZN, TSLA, NKE, SBUX |
|
||||
| Financial Services | JPM, GS, V, MA |
|
||||
| Healthcare | JNJ, UNH, PFE, LLY |
|
||||
| Energy | XOM, CVX, COP |
|
||||
| Communication Services | NFLX, DIS, T |
|
||||
| Industrials | CAT, BA, UPS |
|
||||
| Consumer Defensive | PG, KO, WMT |
|
||||
| Real Estate | AMT, PLD |
|
||||
| Utilities | NEE, DUK |
|
||||
|
||||
46 competitor relationships are defined (direct rivals, same-sector peers, overlapping products, supply chain adjacencies) so the system can propagate signals — e.g., if a semiconductor shortage hits one chipmaker, the system assesses exposure for its competitors and supply chain partners.
|
||||
|
||||
---
|
||||
|
||||
## The Three Signal Layers
|
||||
|
||||
Think of these as three analysts sitting at the same desk, each watching a different feed:
|
||||
|
||||
### Layer 1 — Company-Specific Intelligence
|
||||
|
||||
The bread and butter. Every news article and filing about a specific company gets scored for sentiment, impact magnitude, and time horizon. These signals are weighted by recency (yesterday's earnings matter more than last month's), source credibility, and novelty (the fifth article repeating the same news adds less information than the first).
|
||||
|
||||
Trend summaries roll up across five windows: intraday, 1 day, 7 days, 30 days, and 90 days — giving both a "what's happening right now" and a "what's the longer arc" view.
|
||||
|
||||
### Layer 2 — Macro & Geopolitical
|
||||
|
||||
Global events (trade wars, rate decisions, geopolitical crises, commodity shocks) are classified by impact type and severity. Each company has an exposure profile — geographic revenue mix, supply chain regions, commodity dependencies — that maps macro events down to company-level impact scores.
|
||||
|
||||
A tariff announcement on Chinese imports doesn't affect all 50 companies equally. Apple with its Chinese manufacturing exposure gets a higher impact score than Procter & Gamble with largely domestic supply chains.
|
||||
|
||||
### Layer 3 — Competitive & Historical Patterns
|
||||
|
||||
The system mines its own history: when this type of catalyst (say, an earnings beat) happened to this company in the past, what happened to the stock? What happened to its competitors? If NVIDIA reports a blowout quarter, does AMD tend to sell off or rally in sympathy?
|
||||
|
||||
This layer also tracks major corporate actions (M&A, restructurings, leadership changes) and propagates their implications across the competitive web.
|
||||
|
||||
**Safety rule:** The system never trades on macro or competitive signals alone. If there's no company-specific evidence supporting the thesis, the recommendation is downgraded to informational only.
|
||||
|
||||
---
|
||||
|
||||
## How a Trade Happens
|
||||
|
||||
Here's the chain from "news article published" to "paper order placed":
|
||||
|
||||
1. **Ingestion** — The article is fetched, deduplicated, and stored.
|
||||
2. **Parsing** — Raw HTML is cleaned, boilerplate is stripped, quality is scored.
|
||||
3. **Extraction** — The AI model reads the cleaned text and produces structured JSON: tickers mentioned, sentiment, catalysts, key facts, risks.
|
||||
4. **Aggregation** — The new extraction is merged into rolling trend summaries for each mentioned company. Confidence, contradiction, and evidence depth are recalculated.
|
||||
5. **Recommendation** — If the trend passes quality filters (enough evidence, high enough confidence, low enough contradiction, not stale), a BUY or SELL recommendation is generated with a written thesis.
|
||||
6. **Risk checks** — The trading engine asks: Is the circuit breaker tripped? Is the market open? Do I already have too many positions? Is this sector already overweight? Are earnings in the next 48 hours?
|
||||
7. **Position sizing** — Dollar amount is computed from confidence, portfolio heat, and the current risk tier (conservative / moderate / aggressive — auto-adjusted based on trailing performance).
|
||||
8. **Execution** — The order goes to Alpaca's paper-trading API. Stop-loss and take-profit levels are set automatically based on the stock's recent volatility.
|
||||
9. **Monitoring** — Open positions are tracked with trailing stops. If a position declines past its stop, it's closed. If it hits the take-profit target, it's closed.
|
||||
10. **Scoring** — Days later, the prediction is evaluated against the actual price move. Did the call go the right way? Did the confidence track reality?
|
||||
|
||||
---
|
||||
|
||||
## Risk Management (Built In, Not Bolted On)
|
||||
|
||||
- **Circuit breakers** — If daily losses exceed a threshold or a single position loses too much, all trading halts automatically.
|
||||
- **Position caps** — No single position can consume more than a set percentage of the portfolio.
|
||||
- **Sector concentration limits** — The system won't pile into one sector even if all signals are bullish.
|
||||
- **Correlation awareness** — New positions are rejected if they'd push portfolio correlation too high.
|
||||
- **Earnings blackout** — Position sizes are reduced or skipped entirely within 48 hours of an earnings announcement.
|
||||
- **Reserve pool** — Profits are partially siphoned into an emergency liquidity reserve.
|
||||
- **Risk tier auto-adjustment** — The system evaluates its own Sharpe ratio, drawdown, and win rate daily and shifts between conservative, moderate, and aggressive modes.
|
||||
|
||||
---
|
||||
|
||||
## Self-Grading: The Validation Loop
|
||||
|
||||
Most trading systems tell you their view. Few systematically check whether that view was right.
|
||||
|
||||
Stonks Oracle captures every prediction as an immutable snapshot — the thesis, the confidence, the price at the time, the evidence cited. Then it waits. After the prediction's time horizon elapses (1 day, 7 days, 30 days), it compares the predicted direction against the actual price movement and computes:
|
||||
|
||||
- **Win rate** — What fraction of directional calls were correct?
|
||||
- **Calibration** — When the system says "70% confident bullish," does the stock actually go up ~70% of the time? (If it only goes up 50% of the time, the system is overconfident.)
|
||||
- **Information coefficient** — Does the system's score have any linear correlation with actual returns?
|
||||
- **Excess return vs. SPY** — Is it adding alpha, or would you be better off in an index fund?
|
||||
- **Source attribution** — Which news sources and signal types actually contribute to correct predictions? Which are noise?
|
||||
|
||||
If model quality drops below defined thresholds, a safety gate prevents the system from upgrading recommendations from "informational" to "paper eligible" — it forces itself to the sidelines until accuracy recovers.
|
||||
|
||||
---
|
||||
|
||||
## The Dashboard
|
||||
|
||||
A web-based interface lets you see everything the system sees:
|
||||
|
||||
- **Home** — Portfolio value, daily P&L, risk tier, active alerts.
|
||||
- **Companies** — The tracked universe with current trend summaries and signal strength.
|
||||
- **Documents** — Every ingested article and filing, with the AI's structured extraction visible.
|
||||
- **Trends** — Per-company trend charts across all time windows, with evidence chains you can click through.
|
||||
- **Recommendations** — Active and historical recommendations with full theses and risk classifications.
|
||||
- **Trading** — The engine's status: open positions, reserve pool, circuit breaker state, portfolio heat map.
|
||||
- **Orders & Positions** — Full trade blotter with execution details.
|
||||
- **Macro Events** — Global event timeline showing what the system is tracking at the geopolitical level.
|
||||
- **Reports** — AI-generated daily and weekly performance summaries.
|
||||
- **Model Performance** — Calibration curves, win rate trends, source reliability scores.
|
||||
- **SQL Explorer** — Ad-hoc queries against the full analytical data warehouse, with a chart builder.
|
||||
|
||||
---
|
||||
|
||||
## What It Is Not
|
||||
|
||||
- **Not a live trading system (yet).** All trades are paper trades through Alpaca's sandbox. The architecture supports live execution, but safety gates and validation must demonstrate consistent edge before real money is at risk.
|
||||
- **Not a black box.** Every recommendation includes a full thesis, every trade has a decision trace, every prediction links back to the specific evidence that drove it.
|
||||
- **Not a prediction guarantee.** Markets are hard. The system's value is in disciplined synthesis, consistent process, and honest self-measurement — not in claiming to always be right.
|
||||
|
||||
---
|
||||
|
||||
## Where It's Headed
|
||||
|
||||
Active development is upgrading the signal math from rule-based heuristics to probabilistic Bayesian inference — running both approaches in parallel, comparing their verdicts, and using the disagreements as training signals for continuous improvement. The goal is a system that not only reads the market but learns from its own track record which types of evidence, in which market regimes, actually predict future price moves.
|
||||
Reference in New Issue
Block a user