stonks-oracle/.kiro/specs/model-validation-calibration/tasks.md

Implementation Plan: Model Validation, Calibration, and Signal Quality

Overview

Add a closed-loop model validation layer to Stonks Oracle: prediction snapshot capture, outcome evaluation, calibration/IC metrics, source/catalyst/layer attribution, Bayesian source reliability, a quality gate for live trading, 7 new API endpoints, an upgraded OpsModel dashboard, and backtest replay integration. Implementation follows the four-phase priority order from the spec, with each phase building on the previous one.

Tasks

• 1. Database migration 035 — schema foundation

  • 1.1 Create infra/migrations/035_model_validation.sql with all tables, indexes, and views
    • Create prediction_snapshots table with all columns from design (id UUID PK, generated_at, ticker, window, horizon, direction, action, mode, strength, confidence, contradiction, p_bull, p_bear, score_company, score_macro, score_competitive, evidence_count, unique_source_count, duplicate_evidence_count, price_at_prediction, spy_price_at_prediction, sector_etf_price_at_prediction, metadata JSONB, created_at)
    • Create prediction_outcomes table with FK to prediction_snapshots (id UUID PK, prediction_id, evaluated_at, horizon, future_price, future_return, spy_future_price, spy_return, sector_etf_future_price, sector_etf_return, excess_return_vs_spy, excess_return_vs_sector, direction_correct, profitable, metadata JSONB, created_at)
    • Create signal_evidence_links table with FK to prediction_snapshots (id UUID PK, prediction_id, document_id, signal_id, ticker, source, source_type, catalyst_type, sentiment, impact, extraction_confidence, weight, is_duplicate, canonical_evidence_key, contribution_score, metadata JSONB, created_at)
    • Create model_metric_snapshots table (id UUID PK, generated_at, lookback_window, horizon, prediction_count, win_rate, directional_accuracy, information_coefficient, rank_information_coefficient, avg_return, avg_excess_return_vs_spy, avg_excess_return_vs_sector, calibration_error, brier_score, buy_win_rate, sell_win_rate, hold_win_rate, metadata JSONB, created_at)
    • Create indexes on prediction_snapshots (ticker, generated_at, horizon), prediction_outcomes (prediction_id, horizon, evaluated_at), signal_evidence_links (prediction_id, document_id, ticker), model_metric_snapshots (generated_at, lookback_window, horizon)
    • Create v_prediction_performance view joining prediction_snapshots with prediction_outcomes
    • Create v_source_performance view joining signal_evidence_links with prediction_snapshots and prediction_outcomes
    • Requirements: 16.1, 16.2, 16.3, 16.4, 16.5, 16.6, 14.1, 14.2, 14.3, 14.4

• 2. Phase 1 — Prediction capture, outcome evaluation, core metrics, and dashboard API

  • 2.1 Implement Prediction Snapshot Writer (services/validation/prediction_snapshot.py)

    • Create services/validation/__init__.py
    • Define SECTOR_ETF_MAP, EVALUATION_HORIZONS, MAX_SINGLE_DOCUMENT_WEIGHT constants
    • Implement PredictionSnapshot and SignalEvidenceLink dataclasses
    • Implement compute_canonical_evidence_key(title, url) — SHA256 of normalized title + normalized URL (lowercase, strip whitespace for title; lowercase, strip query params for URL)
    • Implement fetch_latest_close_price(pool, ticker) — query most recent close from market_snapshots
    • Implement create_prediction_snapshot(pool, recommendation, trend_summary, evidence_signals, evidence_docs) — fetch prices (ticker, SPY, sector ETF), compute canonical keys, detect duplicates, clamp weights to MAX_SINGLE_DOCUMENT_WEIGHT, compute contribution scores (one-vote-per-canonical-key), persist snapshot + evidence links in a transaction
    • Implement compute_contribution_scores(weights) — each score = weight_i / sum(weights), sums to 1.0
    • Handle NULL prices gracefully (log warning, store NULL, don't fail)
    • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 3.1, 3.2, 3.3, 3.4

  • 2.2 Write property test for canonical evidence key determinism and idempotence

    • Property 4: Canonical Evidence Key Determinism and Normalization Idempotence
    • Test that same (title, url) always produces same key
    • Test that normalizing already-normalized input produces same key
    • Validates: Requirements 2.3, 17.4

  • 2.3 Write property test for contribution score sum-to-one and range

    • Property 7: Contribution Score Sum-to-One and Range
    • Test that all scores in [0.0, 1.0] and sum to 1.0 (within 1e-9 tolerance)
    • Test that empty input returns empty list
    • Validates: Requirements 2.5, 17.7

  • 2.4 Implement Outcome Evaluator (services/validation/outcome_evaluator.py)

    • Define PredictionOutcome dataclass and HORIZON_DURATIONS mapping
    • Implement evaluate_matured_predictions(pool) — find snapshots where horizon elapsed and outcome not recorded, evaluate each
    • Implement evaluate_single_prediction(pool, snapshot, horizon) — fetch future price at horizon endpoint, compute future_return, SPY return, sector ETF return, excess returns, direction_correct, profitable; return None if future price unavailable
    • Evaluate across all 5 horizons: 1h, 6h, 1d, 7d, 30d
    • Skip horizons where future price is unavailable (retry next run)
    • Store results in prediction_outcomes table
    • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 4.10

  • 2.5 Implement Metrics Engine (services/validation/metrics.py)

    • Define CONFIDENCE_BUCKETS, LOOKBACK_WINDOWS constants
    • Define CalibrationBucket and ModelMetricSnapshot dataclasses
    • Implement compute_calibration_error(confidences, outcomes) — group into 5 confidence buckets, compute ECE as weighted average of |avg_conf - win_rate|, flag miscalibrated buckets (|diff| > 0.15)
    • Implement compute_brier_score(p_bulls, outcomes) — mean((p_bull - outcome)^2)
    • Implement compute_information_coefficient(scores, returns) — Pearson correlation, return None when < 30 data points
    • Implement compute_rank_information_coefficient(scores, returns) — Spearman rank correlation, return None when < 30 data points
    • Implement compute_contribution_scores(weights) — weight_i / sum(weights), sums to 1.0
    • Implement benchmark metrics: average excess return vs SPY, vs sector ETF, hit rate improvement
    • Implement compute_and_store_metric_snapshots(pool) — compute for all lookback/horizon combinations (4 lookbacks × 5 horizons), persist to model_metric_snapshots
    • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 6.1, 6.2, 6.3, 6.4, 6.5, 9.1, 9.2, 9.3, 9.4, 10.1, 10.2, 10.3, 10.4, 10.5

  • 2.6 Write property test for ECE range and round-trip

    • Property 1: Calibration Error Range and Round-Trip
    • Test ECE in [0.0, 1.0] for all valid distributions
    • Test ECE = 0.0 when every bucket's win rate matches avg confidence
    • Validates: Requirements 5.1, 5.3, 17.1

  • 2.7 Write property test for Brier score range and perfect prediction

    • Property 2: Brier Score Range and Perfect Prediction
    • Test Brier in [0.0, 1.0] for all valid (p_bull, outcome) pairs
    • Test Brier = 0.0 when all predictions perfectly correct
    • Validates: Requirements 5.4, 17.2

  • 2.8 Write property test for IC range and perfect correlation

    • Property 3: Information Coefficient Range and Perfect Correlation
    • Test IC in [-1.0, 1.0] for all valid (score, return) pairs with ≥30 elements
    • Test IC = 1.0 for perfectly positively correlated data
    • Validates: Requirements 6.1, 6.2, 17.3

  • 2.9 Implement Dashboard API endpoints in services/api/app.py

    • Add /api/validation/summary GET — return latest model_metric_snapshot + gate status
    • Add /api/validation/calibration GET — return calibration table with buckets
    • Add /api/validation/ic-by-horizon GET — return IC and Rank IC per horizon
    • Add /api/validation/gate-status GET — return quality gate evaluation detail
    • All endpoints accept optional lookback (default "30d") and horizon (default "7d") query params
    • Requirements: 12.1, 12.2, 12.3, 12.7

  • 2.10 Add frontend validation API hooks in frontend/src/api/hooks.ts

    • Add useValidationSummary(lookback?, horizon?) hook for /api/validation/summary
    • Add useValidationCalibration(lookback?, horizon?) hook for /api/validation/calibration
    • Add useValidationICByHorizon(lookback?) hook for /api/validation/ic-by-horizon
    • Add useValidationGateStatus() hook for /api/validation/gate-status
    • Requirements: 12.1, 12.2, 12.3, 12.7

  • 2.11 Upgrade OpsModel page (frontend/src/pages/OpsModel.tsx) — Phase 1 dashboard

    • Add tabbed layout: existing "Extraction Performance" tab + new "Model Validation" tab
    • Add summary cards: prediction count, win rate, directional accuracy, IC, Rank IC, Brier score, ECE, avg excess return vs SPY, gate status
    • Add calibration table with confidence buckets, avg confidence, observed win rate, count, miscalibration flag
    • Highlight miscalibrated buckets (|avg_confidence - observed_win_rate| > 0.15) with warning indicator
    • Add IC-by-horizon table showing IC and Rank IC for each horizon
    • Add gate status indicator (pass/fail with threshold details)
    • Requirements: 12.1, 12.2, 12.3, 12.7, 12.8, 12.9

• 3. Checkpoint — Phase 1 verification

  • Ensure all tests pass, ask the user if questions arise.

• 4. Phase 2 — Attribution engine and source/catalyst truth tables

  • 4.1 Implement Attribution Engine (services/validation/attribution.py)

    • Define SourceAttribution, CatalystAttribution, LayerAttribution dataclasses
    • Implement compute_source_attribution(pool, lookback_days, horizon) — join signal_evidence_links with prediction_outcomes, group by source; compute prediction count, avg weight, avg contribution score, win rate, avg future return, avg excess return vs SPY, IC, duplicate rate
    • Implement compute_catalyst_attribution(pool, lookback_days, horizon) — same metrics grouped by catalyst_type
    • Implement compute_layer_attribution(pool, lookback_days, horizon) — compute per-layer (company, macro, competitive) avg contribution %, dominant win rate (layer > 30% contribution), dominant IC
    • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7

  • 4.2 Implement Calibration Engine (services/validation/calibration.py)

    • Implement compute_source_reliability(observed_win_rate, sample_count, prior_strength=30) — Bayesian shrinkage: 0.5 + (n / (n + 30)) * (observed_win_rate - 0.5); return 0.5 when n=0
    • Implement compute_adjusted_evidence_weight(base_weight, reliability) — base_weight * (0.5 + reliability), clamped to [0.1, 2.0]
    • Implement update_source_reliabilities(pool) — recompute from latest outcomes, update source_accuracy table
    • Requirements: 8.1, 8.2, 8.3, 8.4, 8.5

  • 4.3 Write property test for source reliability Bayesian shrinkage bounds and convergence

    • Property 5: Source Reliability Bayesian Shrinkage Bounds and Convergence
    • Test reliability in [0.0, 1.0] for all valid inputs
    • Test reliability = 0.5 when sample_count = 0
    • Test reliability approaches observed_win_rate as sample_count → ∞
    • Validates: Requirements 8.1, 8.2, 17.5

  • 4.4 Add attribution API endpoints in services/api/app.py

    • Add /api/validation/attribution/sources GET — return per-source performance metrics
    • Add /api/validation/attribution/catalysts GET — return per-catalyst performance metrics
    • Add /api/validation/attribution/layers GET — return per-layer performance metrics
    • All endpoints accept optional lookback (default "30d") and horizon (default "7d") query params
    • Requirements: 12.4, 12.5, 12.6

  • 4.5 Add frontend attribution hooks in frontend/src/api/hooks.ts

    • Add useValidationAttributionSources(lookback?, horizon?) hook
    • Add useValidationAttributionCatalysts(lookback?, horizon?) hook
    • Add useValidationAttributionLayers(lookback?, horizon?) hook
    • Requirements: 12.4, 12.5, 12.6

  • 4.6 Extend OpsModel page with attribution tables

    • Add source performance table (source, win rate, IC, avg return, duplicate rate)
    • Add catalyst truth table (catalyst type, win rate, avg return, IC)
    • Add layer attribution table (company/macro/competitive contribution %, dominant win rate, IC)
    • Requirements: 12.4, 12.5, 12.6, 12.8

• 5. Checkpoint — Phase 2 verification

  • Ensure all tests pass, ask the user if questions arise.

• 6. Phase 3 — Quality gate, recommendation enhancements, and pipeline wiring

  • 6.1 Implement Quality Gate (services/trading/model_quality_gate.py)

    • Define QualityGateConfig dataclass with default thresholds (min_prediction_count=100, min_ic=0.03, min_win_rate=0.53, max_ece=0.15, min_excess_return_vs_spy=0.0, max_snapshot_age_hours=24)
    • Define GateThresholdResult and QualityGateResult dataclasses
    • Implement evaluate_quality_gate(pool, config) — read most recent model_metric_snapshot (30d lookback, 7d horizon), evaluate each threshold, store result in risk_configs under 'model_quality_gate' key
    • Implement load_gate_config_from_db(pool) — load thresholds from risk_configs with defaults
    • Default to paper-only mode when no snapshots exist or snapshot is stale (>24h)
    • Log gate evaluation result with threshold pass/fail details
    • Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7

  • 6.2 Write property test for quality gate determinism and threshold monotonicity

    • Property 6: Quality Gate Determinism and Threshold Monotonicity
    • Test same inputs always produce same pass/fail result
    • Test relaxing any threshold never causes a previously passing gate to fail
    • Validates: Requirements 11.1, 17.6

  • 6.3 Wire Quality Gate into aggregation cycle (services/aggregation/worker.py)

    • Call evaluate_quality_gate at the start of each aggregation cycle
    • When gate fails, force all recommendations to paper mode
    • Log gate status at cycle start
    • Requirements: 11.2, 11.3

  • 6.4 Wire Prediction Snapshot Writer into recommendation engine

    • After recommendation is generated in services/recommendation/eligibility.py or the calling code, call create_prediction_snapshot to capture the prediction state
    • Pass recommendation, trend_summary, evidence signals, and evidence docs
    • Handle snapshot creation failure gracefully (log error, don't block recommendation)
    • Requirements: 1.1, 1.6

  • 6.5 Enhance recommendation display on frontend

    • Update frontend/src/pages/RecommendationDetail (or relevant recommendation display component) to show:
      • Original confidence alongside calibrated confidence (historical win rate for that bucket)
      • Historical win rate for similar confidence levels
      • Evidence count, unique evidence count, duplicate evidence count
      • Source reliability indicator for primary contributing sources
      • Live eligibility status with reason (gate passed or which threshold failed)
    • Add warning badge when duplicate evidence count > 20% of total evidence count
    • Add warning badge when primary source reliability < 0.4
    • Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7

• 7. Checkpoint — Phase 3 verification

  • Ensure all tests pass, ask the user if questions arise.

• 8. Phase 4 — Backtest replay integration and unit tests

  • 8.1 Add validation mode to BacktestReplay (services/trading/backtest_replay.py)

    • Add validation_mode: bool = False parameter to BacktestReplay.run()
    • When validation_mode=True, create prediction snapshots for each historical recommendation using only data available at that point in time
    • Evaluate prediction outcomes using market prices from the appropriate future horizon
    • Prevent future data leakage: no market data after prediction generation time used during snapshot creation
    • After backtest completes, trigger model metrics computation over the backtest period, tag snapshots with backtest_id
    • Requirements: 15.1, 15.2, 15.3, 15.4, 15.5

  • 8.2 Write unit tests for prediction snapshot writer (tests/test_model_validation_unit.py)

    • Test canonical evidence key: known title/URL → expected SHA256, empty inputs, unicode
    • Test duplicate detection: 3 docs with 2 sharing a key → 1 marked duplicate
    • Test contribution scores: [0.5, 0.3, 0.2] → [0.5, 0.3, 0.2], single doc → [1.0]
    • Test weight clamping: weight 1.5 → clamped to 1.0
    • Requirements: 1.1, 2.3, 2.4, 2.5, 3.3

  • 8.3 Write unit tests for outcome evaluator (tests/test_model_validation_unit.py)

    • Test future return computation: price 100→110 → 0.10, price 100→90 → -0.10
    • Test direction_correct logic: bullish+positive → true, bullish+negative → false
    • Test profitable logic: buy+positive → true, sell+negative → true
    • Test excess return: ticker 10%, SPY 5% → excess 5%
    • Requirements: 4.2, 4.5, 4.6, 4.7

  • 8.4 Write unit tests for metrics engine (tests/test_model_validation_unit.py)

    • Test ECE specific values: perfect calibration → 0.0, all overconfident → positive ECE
    • Test Brier score: all correct at p=1.0 → 0.0, all wrong at p=1.0 → 1.0
    • Test IC: perfect correlation → 1.0, anti-correlation → -1.0, < 30 → None
    • Requirements: 5.3, 5.4, 6.1, 6.2, 6.5

  • 8.5 Write unit tests for calibration engine (tests/test_model_validation_unit.py)

    • Test source reliability: n=0 → 0.5, n=1000 with wr=0.8 → ≈0.8, n=30 with wr=0.7 → 0.6
    • Test adjusted evidence weight: reliability=0.5 → base*1.0, clamping to [0.1, 2.0]
    • Requirements: 8.1, 8.2, 8.3

  • 8.6 Write unit tests for quality gate (tests/test_model_validation_unit.py)

    • Test all thresholds met → pass
    • Test one threshold failed → fail with reason
    • Test fail-safe: no snapshots → paper-only, stale snapshot → paper-only
    • Requirements: 11.1, 11.6

  • 8.7 Write frontend tests for validation dashboard (frontend/src/test/pages.test.tsx)

    • Add MSW mock handlers for /api/validation/summary, /api/validation/calibration, /api/validation/gate-status
    • Test OpsModel page renders validation tab with summary cards
    • Test calibration table renders buckets with miscalibration warning
    • Test gate status indicator renders pass/fail
    • Requirements: 12.8, 12.9

• 9. Final checkpoint — Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

Notes

• Tasks marked with * are optional and can be skipped for faster MVP
• Each task references specific requirements for traceability
• Checkpoints ensure incremental validation after each phase
• Property tests validate the 7 universal correctness properties from the design document
• Unit tests validate specific examples, edge cases, and integration points
• The design uses Python for backend and TypeScript for frontend — no language selection needed
• Migration number is 035 (existing migrations go up to 034)
• All new service modules go under services/validation/ except the quality gate which goes in services/trading/
• The 7 new API endpoints are added to the existing services/api/app.py
• Frontend hooks follow existing patterns in frontend/src/api/hooks.ts
• Phase 1 delivers the core feedback loop (capture → evaluate → measure → display)
• Phase 2 adds attribution depth (which sources/catalysts/layers work best)
• Phase 3 adds safety (quality gate) and UX (recommendation warnings)
• Phase 4 adds historical analysis (backtest validation mode) and comprehensive tests