feat: model validation, calibration, and signal quality layer
ci/woodpecker/push/test Pipeline failed
ci/woodpecker/push/build-1 unknown status
ci/woodpecker/push/build-3 unknown status
ci/woodpecker/push/build-2 unknown status
ci/woodpecker/push/finalize unknown status
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
ci/woodpecker/push/test Pipeline failed
ci/woodpecker/push/build-1 unknown status
ci/woodpecker/push/build-3 unknown status
ci/woodpecker/push/build-2 unknown status
ci/woodpecker/push/finalize unknown status
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
- Migration 035: prediction_snapshots, prediction_outcomes, signal_evidence_links, model_metric_snapshots tables + SQL views - Prediction snapshot writer with canonical evidence keys, duplicate detection, contribution scores - Outcome evaluator across 5 horizons (1h, 6h, 1d, 7d, 30d) - Metrics engine: ECE, Brier score, IC, Rank IC, benchmark comparison - Attribution engine: per-source, per-catalyst, per-layer performance - Calibration engine: Bayesian shrinkage source reliability - Quality gate for live trading eligibility with configurable thresholds - 7 new /api/validation/* endpoints - Upgraded OpsModel dashboard with validation tab - Enhanced recommendation display with calibration context - Backtest replay validation mode - 86 Python tests (unit + property-based), 179 frontend tests passing
This commit is contained in:
@@ -0,0 +1,176 @@
|
||||
-- Migration 035: Model Validation, Calibration, and Signal Quality
|
||||
-- Creates tables for prediction snapshots, outcomes, evidence links, and metric snapshots
|
||||
-- Plus views for prediction performance and source performance analysis
|
||||
|
||||
-- ============================================================================
|
||||
-- Table: prediction_snapshots
|
||||
-- Immutable snapshot of a prediction at generation time
|
||||
-- ============================================================================
|
||||
CREATE TABLE IF NOT EXISTS prediction_snapshots (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
generated_at TIMESTAMPTZ NOT NULL,
|
||||
ticker VARCHAR(20) NOT NULL,
|
||||
window VARCHAR(20) NOT NULL,
|
||||
horizon VARCHAR(20) NOT NULL,
|
||||
direction VARCHAR(20) NOT NULL,
|
||||
action VARCHAR(20) NOT NULL,
|
||||
mode VARCHAR(30) NOT NULL,
|
||||
strength FLOAT NOT NULL,
|
||||
confidence FLOAT NOT NULL,
|
||||
contradiction FLOAT NOT NULL DEFAULT 0.0,
|
||||
p_bull FLOAT,
|
||||
p_bear FLOAT,
|
||||
score_company FLOAT NOT NULL DEFAULT 0.0,
|
||||
score_macro FLOAT NOT NULL DEFAULT 0.0,
|
||||
score_competitive FLOAT NOT NULL DEFAULT 0.0,
|
||||
evidence_count INTEGER NOT NULL DEFAULT 0,
|
||||
unique_source_count INTEGER NOT NULL DEFAULT 0,
|
||||
duplicate_evidence_count INTEGER NOT NULL DEFAULT 0,
|
||||
price_at_prediction FLOAT,
|
||||
spy_price_at_prediction FLOAT,
|
||||
sector_etf_price_at_prediction FLOAT,
|
||||
metadata JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_pred_snap_ticker ON prediction_snapshots(ticker);
|
||||
CREATE INDEX IF NOT EXISTS idx_pred_snap_generated ON prediction_snapshots(generated_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_pred_snap_horizon ON prediction_snapshots(horizon);
|
||||
|
||||
-- ============================================================================
|
||||
-- Table: prediction_outcomes
|
||||
-- Realized outcome for a prediction at a specific horizon
|
||||
-- ============================================================================
|
||||
CREATE TABLE IF NOT EXISTS prediction_outcomes (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
prediction_id UUID NOT NULL REFERENCES prediction_snapshots(id),
|
||||
evaluated_at TIMESTAMPTZ NOT NULL,
|
||||
horizon VARCHAR(20) NOT NULL,
|
||||
future_price FLOAT,
|
||||
future_return FLOAT,
|
||||
spy_future_price FLOAT,
|
||||
spy_return FLOAT,
|
||||
sector_etf_future_price FLOAT,
|
||||
sector_etf_return FLOAT,
|
||||
excess_return_vs_spy FLOAT,
|
||||
excess_return_vs_sector FLOAT,
|
||||
direction_correct BOOLEAN,
|
||||
profitable BOOLEAN,
|
||||
metadata JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_pred_out_prediction ON prediction_outcomes(prediction_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_pred_out_horizon ON prediction_outcomes(horizon);
|
||||
CREATE INDEX IF NOT EXISTS idx_pred_out_evaluated ON prediction_outcomes(evaluated_at);
|
||||
|
||||
-- ============================================================================
|
||||
-- Table: signal_evidence_links
|
||||
-- Link between a prediction and a contributing evidence document
|
||||
-- ============================================================================
|
||||
CREATE TABLE IF NOT EXISTS signal_evidence_links (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
prediction_id UUID NOT NULL REFERENCES prediction_snapshots(id),
|
||||
document_id VARCHAR(200),
|
||||
signal_id VARCHAR(200),
|
||||
ticker VARCHAR(20),
|
||||
source VARCHAR(200),
|
||||
source_type VARCHAR(50),
|
||||
catalyst_type VARCHAR(50),
|
||||
sentiment VARCHAR(20),
|
||||
impact FLOAT,
|
||||
extraction_confidence FLOAT,
|
||||
weight FLOAT,
|
||||
is_duplicate BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
canonical_evidence_key VARCHAR(64),
|
||||
contribution_score FLOAT,
|
||||
metadata JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sig_ev_prediction ON signal_evidence_links(prediction_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_sig_ev_document ON signal_evidence_links(document_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_sig_ev_ticker ON signal_evidence_links(ticker);
|
||||
|
||||
-- ============================================================================
|
||||
-- Table: model_metric_snapshots
|
||||
-- Aggregate model quality metrics for a lookback/horizon combination
|
||||
-- ============================================================================
|
||||
CREATE TABLE IF NOT EXISTS model_metric_snapshots (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
generated_at TIMESTAMPTZ NOT NULL,
|
||||
lookback_window VARCHAR(20) NOT NULL,
|
||||
horizon VARCHAR(20) NOT NULL,
|
||||
prediction_count INTEGER NOT NULL DEFAULT 0,
|
||||
win_rate FLOAT,
|
||||
directional_accuracy FLOAT,
|
||||
information_coefficient FLOAT,
|
||||
rank_information_coefficient FLOAT,
|
||||
avg_return FLOAT,
|
||||
avg_excess_return_vs_spy FLOAT,
|
||||
avg_excess_return_vs_sector FLOAT,
|
||||
calibration_error FLOAT,
|
||||
brier_score FLOAT,
|
||||
buy_win_rate FLOAT,
|
||||
sell_win_rate FLOAT,
|
||||
hold_win_rate FLOAT,
|
||||
metadata JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_model_snap_generated ON model_metric_snapshots(generated_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_model_snap_lookback ON model_metric_snapshots(lookback_window);
|
||||
CREATE INDEX IF NOT EXISTS idx_model_snap_horizon ON model_metric_snapshots(horizon);
|
||||
|
||||
-- ============================================================================
|
||||
-- View: v_prediction_performance
|
||||
-- Joins prediction snapshots with outcomes for flat analysis
|
||||
-- ============================================================================
|
||||
CREATE OR REPLACE VIEW v_prediction_performance AS
|
||||
SELECT
|
||||
ps.ticker,
|
||||
ps.direction,
|
||||
ps.action,
|
||||
ps.confidence,
|
||||
ps.strength,
|
||||
ps.contradiction,
|
||||
ps.p_bull,
|
||||
ps.score_company,
|
||||
ps.score_macro,
|
||||
ps.score_competitive,
|
||||
ps.evidence_count,
|
||||
ps.unique_source_count,
|
||||
ps.duplicate_evidence_count,
|
||||
ps.price_at_prediction,
|
||||
po.future_return,
|
||||
po.excess_return_vs_spy,
|
||||
po.excess_return_vs_sector,
|
||||
po.direction_correct,
|
||||
po.profitable,
|
||||
po.horizon,
|
||||
ps.generated_at,
|
||||
po.evaluated_at
|
||||
FROM prediction_snapshots ps
|
||||
JOIN prediction_outcomes po ON po.prediction_id = ps.id;
|
||||
|
||||
-- ============================================================================
|
||||
-- View: v_source_performance
|
||||
-- Joins evidence links with snapshots and outcomes for source attribution
|
||||
-- ============================================================================
|
||||
CREATE OR REPLACE VIEW v_source_performance AS
|
||||
SELECT
|
||||
sel.source,
|
||||
sel.source_type,
|
||||
sel.catalyst_type,
|
||||
sel.sentiment,
|
||||
sel.weight,
|
||||
sel.contribution_score,
|
||||
sel.is_duplicate,
|
||||
po.direction_correct,
|
||||
po.future_return,
|
||||
po.excess_return_vs_spy,
|
||||
po.horizon,
|
||||
ps.generated_at
|
||||
FROM signal_evidence_links sel
|
||||
JOIN prediction_snapshots ps ON ps.id = sel.prediction_id
|
||||
JOIN prediction_outcomes po ON po.prediction_id = sel.prediction_id;
|
||||
Reference in New Issue
Block a user