stonks-oracle/.kiro/specs/signal-math-upgrade/tasks.md

Implementation Plan: Signal Math Upgrade

Overview

Upgrade the Stonks Oracle signal processing pipeline from deterministic heuristic formulas to a probabilistic, regime-aware, and adaptive mathematical framework. Implementation proceeds in layers: foundations (config, schemas, new modules), then each pipeline stage (scoring → trend assembly → macro → competitive → projection → recommendation), then integration wiring, and finally testing. All changes are gated behind the probabilistic_scoring_enabled feature flag.

Tasks

• 1. Foundation: Configuration and schema extensions

  • 1.1 Extend ScoringConfig with probabilistic parameters in services/aggregation/scoring.py

    • Add probabilistic: bool = False toggle field
    • Add sigmoid gate parameters: sigmoid_steepness, sigmoid_midpoint
    • Add information gain parameters: info_gain_lambda, info_gain_max, default_base_rate
    • Add adaptive decay parameters: adaptive_decay_impact_scale, adaptive_decay_surprise_scale, adaptive_decay_market_scale
    • Add regime multiplier parameters: regime_return_weight, regime_volume_weight, regime_multiplier_max
    • All new fields must have defaults matching the design document values
    • Requirements: 2.5, 3.1, 5.1, 6.3, 16.1

  • 1.2 Extend SignalWeight and WeightedSignal dataclasses in services/aggregation/scoring.py

    • Add optional fields to SignalWeight: sigmoid_gate, info_gain_factor, source_accuracy_factor, regime_multiplier
    • Add optional fields to WeightedSignal: info_gain_factor, source_accuracy_factor, adaptive_half_life
    • All new fields must have defaults (None or 1.0) for backward compatibility
    • Requirements: 16.1, 2.5, 3.3, 4.2

  • 1.3 Extend TrendSummary Pydantic model in services/shared/schemas.py

    • Add optional fields: p_bull, alpha, beta_param, bayesian_confidence, entropy, regime, pipeline_mode
    • pipeline_mode defaults to "heuristic"; all others default to None
    • Requirements: 16.1, 1.6, 9.6

  • 1.4 Extend Recommendation model in services/shared/schemas.py (or services/recommendation/eligibility.py)

    • Add optional fields: expected_value, p_bull, pipeline_mode
    • pipeline_mode defaults to "heuristic"; all others default to None
    • Requirements: 16.1, 14.5

  • 1.5 Add probabilistic_scoring_enabled feature flag support in services/shared/config.py

    • Read probabilistic_scoring_enabled from risk_configs.config JSONB
    • Default to False when key is missing, value is invalid, or DB is unreachable
    • Propagate flag through AggregationConfig dataclass
    • Log which pipeline mode is active at cycle start
    • Requirements: 16.3, 16.4, 16.5, 16.6, 16.7

  • 1.6 Create database migration infra/migrations/034_source_accuracy.sql

    • Create source_accuracy table with columns: id UUID PRIMARY KEY DEFAULT gen_random_uuid(), source_id VARCHAR(200) NOT NULL, accuracy_ratio FLOAT NOT NULL DEFAULT 0.5, sample_count INTEGER NOT NULL DEFAULT 0, last_updated TIMESTAMPTZ, created_at TIMESTAMPTZ
    • Add UNIQUE(source_id) constraint and idx_source_accuracy_source index
    • Requirements: 4.5

• 2. Checkpoint — Verify foundation compiles and existing tests pass

  • Ensure all tests pass, ask the user if questions arise.

• 3. New module: Bayesian Accumulator (services/aggregation/bayesian.py)

  • 3.1 Implement BayesianPosterior dataclass and compute_bayesian_posterior function

    • Create frozen dataclass with fields: p_bull, alpha, beta, log_likelihood, bayesian_confidence, entropy, signal_count
    • Define PRIOR class-level constant for uninformative prior (p_bull=0.5, α=1.0, β=1.0, C=0.0, H=1.0)
    • Implement log-likelihood accumulation: L_t = Σ(w_i · s_i) using weight.combined * sentiment_value
    • Compute P_bull = σ(L_t) via sigmoid function
    • Compute Beta posterior: α = 1 + W_bull, β = 1 + W_bear from positive/negative weight sums
    • Compute Bayesian confidence: C = 1 - 4αβ/(α+β)²
    • Compute Shannon entropy via compute_entropy
    • Return PRIOR for empty signal lists
    • Skip signals with NaN weight or sentiment
    • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6

  • 3.2 Implement compute_entropy function

    • Shannon entropy: H = -p·log₂(p) - (1-p)·log₂(1-p)
    • Return 0.0 for p ≤ 0 or p ≥ 1 (edge cases)
    • Return value in [0, 1] with maximum 1.0 at p=0.5
    • Requirements: 9.1, 9.7

  • 3.3 Write property test for sigmoid gate monotonicity

    • Property 1: Sigmoid Gate Monotonicity
    • Validates: Requirements 2.6, 17.1

  • 3.4 Write property test for Beta posterior evidence accumulation

    • Property 2: Beta Posterior Evidence Accumulation
    • Validates: Requirements 1.3, 17.2

  • 3.5 Write property test for Bayesian confidence symmetry and divergence

    • Property 3: Bayesian Confidence Symmetry and Divergence
    • Validates: Requirements 1.4, 17.3

  • 3.6 Write property test for Bayesian posterior round-trip consistency

    • Property 4: Bayesian Posterior Round-Trip Consistency
    • Validates: Requirements 1.7, 17.7

  • 3.7 Write property test for Shannon entropy range and maximum

    • Property 8: Shannon Entropy Range and Maximum
    • Validates: Requirements 9.7

  • 3.8 Write property test for Bayesian confidence monotonic with agreeing signals

    • Property 13: Bayesian Confidence Monotonic with Agreeing Signals
    • Validates: Requirements 8.6

• 4. New module: Regime Detector (services/aggregation/regime.py)

  • 4.1 Implement MarketRegime enum, RegimeClassification and RegimeConfig dataclasses

    • MarketRegime: TREND_FOLLOWING, PANIC, MEAN_REVERSION, UNCERTAINTY
    • RegimeClassification: regime, trend_indicator, volatility_ratio, bullish_threshold, bearish_threshold, contradiction_penalty_multiplier
    • RegimeConfig: all configurable parameters with defaults from design
    • Requirements: 7.3

  • 4.2 Implement compute_ema and classify_regime functions

    • compute_ema: exponential moving average over last N values
    • classify_regime: compute trend indicator R = sign(EMA_20 - EMA_100) and volatility ratio V_r = σ_20 / σ_100
    • Classification rules: trend-following (R≠0 AND V_r<1.2), panic (V_r>1.5), mean-reversion (R=0 AND V_r<1.0), uncertainty (all other)
    • Adjust thresholds per regime: panic→±0.10, mean-reversion→±0.20, trend-following→±0.15, uncertainty→±0.15 with contradiction multiplier 0.6
    • Default to uncertainty when data is insufficient (<100 days) or σ values are zero
    • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.9

• 5. New module: Source Accuracy Tracker (services/aggregation/source_accuracy.py)

  • 5.1 Implement SourceAccuracy dataclass and database functions
    • SourceAccuracy dataclass with source_id, accuracy_ratio, sample_count, last_updated
    • accuracy_factor property: return 1.0 when sample_count < 10, else 0.5 + accuracy_ratio
    • fetch_source_accuracy: batch fetch from source_accuracy table via asyncpg
    • update_source_accuracy: update accuracy metrics from realized price outcomes
    • Handle DB unreachable: return neutral factor 1.0 for all sources
    • Clamp corrupted accuracy_ratio to [0.0, 1.0]
    • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5

• 6. Checkpoint — Verify new modules compile and unit tests pass

  • Ensure all tests pass, ask the user if questions arise.

• 7. Signal Scorer upgrades (services/aggregation/scoring.py)

  • 7.1 Implement sigmoid confidence gate

    • Add sigmoid_gate(x, steepness, midpoint) function: σ(k·(x - midpoint))
    • When probabilistic=True, replace binary gate with sigmoid gate in compute_signal_weight
    • When probabilistic=False, preserve existing binary gate behavior
    • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5

  • 7.2 Implement information gain surprise weighting

    • Add EVENT_TYPE_BASE_RATES constant dict and DEFAULT_BASE_RATE = 0.1
    • Add compute_info_gain(event_type, lambda_param, max_gain, default_base_rate) function: r = 1 + λ·(-log₂ P(event_type)), clamped to max 3.0
    • Integrate as multiplicative factor in combined weight when probabilistic=True
    • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5

  • 7.3 Implement adaptive recency decay

    • Add compute_adaptive_half_life(base_half_life, impact_score, info_gain_factor, market_multiplier, config) function
    • Compute β_impact, β_surprise, β_market_reaction scaling factors per design
    • τ_i = τ_base · (1 + β_impact) · (1 + β_surprise) · (1 + β_market_reaction)
    • When probabilistic=True, use adaptive half-life in recency_weight; otherwise use fixed
    • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7

  • 7.4 Implement regime multiplier replacing market context multiplier

    • Add compute_regime_multiplier(returns, volumes, config) function
    • Compute z-scores for return and volume, then M_regime = 1 + 0.15·|z_r| + 0.10·|z_v|
    • Clamp to [1.0, 2.5]; default to 1.0 when data unavailable or σ=0
    • When probabilistic=True, use M_regime instead of M_context in combined weight
    • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5

  • 7.5 Integrate source accuracy factor into compute_signal_weight

    • Accept optional source_accuracy_factor parameter
    • When probabilistic=True, multiply into combined weight formula
    • When probabilistic=False, ignore (factor = 1.0)
    • Requirements: 4.2, 4.3

  • 7.6 Update compute_signal_weight to branch on probabilistic flag

    • When probabilistic=True: use sigmoid gate × recency (adaptive) × credibility × (1 + novelty) × info_gain × source_accuracy × regime_multiplier
    • When probabilistic=False: preserve exact current formula (binary gate × recency × credibility × (1 + novelty) × market_context)
    • Populate all new optional fields on SignalWeight and WeightedSignal
    • Requirements: 16.4, 16.5

  • 7.7 Write property test for information gain monotonicity

    • Property 6: Information Gain Monotonicity
    • Validates: Requirements 3.5

  • 7.8 Write property test for adaptive decay lower bound

    • Property 5: Adaptive Decay Lower Bound
    • Validates: Requirements 5.7, 17.4

• 8. Contradiction upgrade (services/aggregation/contradiction.py)

  • 8.1 Implement weighted disagreement entropy contradiction

    • Compute f_pos = W_positive / (W_positive + W_negative) and f_neg = 1 - f_pos
    • Compute H_contradiction = -f_pos·log₂(f_pos) - f_neg·log₂(f_neg)
    • Weight by evidence mass: contradiction_score = H_contradiction · min(1.0, (W_pos + W_neg) / W_threshold)
    • Return 0.0 when only one direction exists
    • Preserve existing ContradictionResult interface
    • When probabilistic=False, preserve existing minority/majority ratio behavior
    • Requirements: 15.1, 15.2, 15.3, 15.4, 15.5, 15.6, 15.7

  • 8.2 Write property test for contradiction entropy monotonicity

    • Property 9: Contradiction Entropy Monotonicity
    • Validates: Requirements 15.7

• 9. Trend Assembly upgrades (services/aggregation/worker.py)

  • 9.1 Integrate Bayesian posterior into trend assembly

    • When probabilistic=True, call compute_bayesian_posterior on merged signals
    • Use Bayesian confidence formula for trend confidence: 0.5 × C_bayesian + 0.25 × F_count + 0.25 × C_avg_credibility - P_contradiction
    • Use entropy-based direction: H>0.9→mixed, P_bull>0.65→bullish, P_bull<0.35→bearish, else neutral
    • Apply regime-adjusted thresholds from RegimeClassification
    • Populate new TrendSummary fields: p_bull, alpha, beta_param, bayesian_confidence, entropy, regime, pipeline_mode
    • Store probabilistic outputs in market_context JSONB under "probabilistic" key
    • When probabilistic=False, preserve exact current heuristic behavior
    • Requirements: 1.1, 1.2, 8.1, 8.2, 8.3, 8.4, 8.5, 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 7.8, 16.4, 16.5

  • 9.2 Wire regime detection into the aggregation cycle

    • Call classify_regime with closing prices and returns for each ticker
    • Pass RegimeClassification to trend assembly for threshold adjustment
    • Default to uncertainty regime when market data is unavailable
    • Persist regime classification in JSONB for auditability
    • Requirements: 7.1, 7.2, 7.3, 7.8, 7.9

• 10. Macro scoring upgrade (services/aggregation/interpolation.py)

  • 10.1 Implement multiplicative macro exposure formula

    • When probabilistic=True, compute S_macro = severity · (1 - Π_k(1 - w_k · O_k)) instead of linear weighted sum
    • Preserve overlap weights: w_geo=0.35, w_supply=0.25, w_commodity=0.25, w_sector=0.15
    • Preserve severity mapping and resilience modifier
    • When probabilistic=False, preserve exact current linear formula
    • Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6

  • 10.2 Implement conditional macro signal integration

    • When probabilistic=True and both company and macro signals exist, apply macro as multiplicative modifier: S_adjusted = S_company · clamp(1 + M_macro · sign_alignment, 0.5, 1.5)
    • When only macro signals exist, fall back to additive behavior with weight 0.3
    • When only company signals exist, use modifier = 1.0
    • Log macro modifier value per ticker
    • When probabilistic=False, preserve current additive merge behavior
    • Requirements: 11.1, 11.2, 11.3, 11.4, 11.5

  • 10.3 Write property test for multiplicative macro exposure monotonicity

    • Property 7: Multiplicative Macro Exposure Monotonicity
    • Validates: Requirements 10.7, 17.5

• 11. Competitive signal upgrade (services/aggregation/signal_propagation.py)

  • 11.1 Implement graph-distance attenuation for competitive signals

    • When probabilistic=True, compute S_transfer = S_source · ρ_historical · e^(-d_network) instead of flat transfer
    • Compute graph distance as shortest path in competitor relationship graph (cap at 3)
    • Use 90-day rolling Pearson correlation for ρ_historical; default to 0.3 (same-sector) or 0.1 (cross-sector) when insufficient data (<30 days)
    • Preserve existing relationship strength threshold (R ≥ 0.2) as pre-filter
    • When probabilistic=False, preserve exact current flat transfer behavior
    • Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 12.7

  • 11.2 Write property test for competitive signal distance attenuation

    • Property 11: Competitive Signal Distance Attenuation
    • Validates: Requirements 12.7

• 12. Projection upgrade (services/aggregation/projection.py)

  • 12.1 Implement exponentially weighted momentum

    • When probabilistic=True, compute M_t = Σ_{k=0}^{K-1} λ^k · ΔS_{t-k} with λ=0.7, K up to 10
    • Normalize by geometric series sum to produce value in [-1, 1]
    • Fall back to current heuristic when fewer than 2 historical cycles available
    • Compute volatility-scaled momentum: M_adj = M_t / max(σ_20, 0.01), clamped to [-2.0, 2.0]
    • When probabilistic=False, preserve exact current simple momentum behavior
    • Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6

  • 12.2 Write property test for exponentially weighted momentum direction

    • Property 10: Exponentially Weighted Momentum Direction
    • Validates: Requirements 13.6, 17.6

• 13. Recommendation upgrade (services/recommendation/eligibility.py)

  • 13.1 Implement expected value recommendation gate

    • When probabilistic=True, compute EV = P_bull · R_up - P_bear · R_down
    • Estimate R_up = strength · σ_20 · √(horizon_days) and R_down = (1 - strength) · σ_20 · √(horizon_days)
    • When EV > threshold (default 0.005), allow recommendation through existing gates
    • When EV ≤ threshold, force recommendation to informational mode
    • Persist EV in risk_checks JSONB of recommendation_evaluations
    • Populate expected_value, p_bull, pipeline_mode on Recommendation model
    • Preserve all existing eligibility gates as additional requirements
    • When probabilistic=False, skip EV gate entirely
    • Requirements: 14.1, 14.2, 14.3, 14.4, 14.5, 14.6

  • 13.2 Write property test for expected value directional consistency

    • Property 12: Expected Value Directional Consistency
    • Validates: Requirements 17.8

• 14. Checkpoint — Verify all pipeline stages compile and existing tests still pass

  • Ensure all tests pass, ask the user if questions arise.

• 15. Integration wiring and feature flag plumbing

  • 15.1 Wire feature flag through the aggregation worker entry point

    • Read probabilistic_scoring_enabled from risk_configs at cycle start in services/aggregation/worker.py
    • Pass flag to ScoringConfig, trend assembly, contradiction, macro, competitive, and projection stages
    • Log pipeline mode at cycle start
    • Ensure flag is read once per cycle (mid-cycle changes take effect next cycle)
    • Requirements: 16.3, 16.6, 16.7

  • 15.2 Wire source accuracy fetch into the scoring pipeline

    • At cycle start, batch-fetch source accuracy for all source IDs in the current signal set
    • Pass source_accuracy_factor to compute_signal_weight for each signal
    • Handle DB errors gracefully (default to 1.0)
    • Requirements: 4.1, 4.2, 4.3

  • 15.3 Wire regime detection into the aggregation cycle

    • Fetch closing prices and returns for each ticker from market data
    • Call classify_regime and pass result to trend assembly and scoring stages
    • Handle missing market data (default to uncertainty regime)
    • Requirements: 7.1, 7.8, 7.9

  • 15.4 Store probabilistic outputs in existing JSONB columns

    • Store Bayesian fields in trend_windows.market_context JSONB under "probabilistic" key
    • Store EV fields in recommendation_evaluations.risk_checks JSONB
    • Store regime classification in trend window JSONB
    • Requirements: 16.2

• 16. Numerical stability and edge case hardening

  • 16.1 Add input validation and edge case guards across all new functions

    • Guard log₂(0) in entropy and information gain computations
    • Floor max(σ_20, 0.01) for momentum volatility scaling
    • Default to uncertainty regime when σ values are zero
    • Return M_regime = 1.0 when z-score σ = 0
    • Skip signals with NaN weight or sentiment
    • Clamp all outputs to documented ranges
    • Requirements: 17.9, 6.4

  • 16.2 Write property test for numerical stability across all formulas

    • Property 14: Numerical Stability Across All Formulas
    • Validates: Requirements 17.9, 6.4

• 17. Unit tests for all new and modified modules

  • 17.1 Write unit tests for Bayesian accumulator (tests/test_bayesian.py)

    • Test uninformative prior (empty signals → P_bull=0.5, α=1, β=1, C=0)
    • Test specific sigmoid gate values (x=0.5→0.5, x=0.2→<0.05, x=0.8→>0.95)
    • Test entropy direction mapping (H>0.9→mixed, P_bull>0.65→bullish, etc.)
    • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5

  • 17.2 Write unit tests for regime detector (tests/test_regime.py)

    • Test specific (R, V_r) → expected regime classification
    • Test threshold adjustments per regime (panic→0.10, mean_reversion→0.20)
    • Test insufficient data fallback to uncertainty
    • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.9

  • 17.3 Write unit tests for source accuracy tracker (tests/test_source_accuracy.py)

    • Test accuracy_factor property: sample_count < 10 → 1.0, else 0.5 + ratio
    • Test corrupted data clamping
    • Requirements: 4.1, 4.2, 4.3

  • 17.4 Write unit tests for signal scoring upgrades (tests/test_signal_math_unit.py)

    • Test info gain clamp (very rare event → factor ≤ 3.0)
    • Test default base rate (unknown event type → 0.1)
    • Test adaptive decay edge cases (all zeros → τ_base, all max → 6×τ_base)
    • Test zero overlap → zero macro impact
    • Test max overlap → ≈severity×0.724
    • Test macro fallback behaviors (only macro → additive, only company → no modifier)
    • Test graph distance cutoff (d>3 → no propagation)
    • Test momentum fallback (<2 cycles → heuristic)
    • Test EV threshold behavior (EV>0.005→proceed, EV≤0.005→informational)
    • Test feature flag behaviors (flag=false→heuristic, flag=true→probabilistic)
    • Requirements: 3.1, 3.4, 5.5, 5.6, 10.3, 10.4, 11.3, 13.3, 14.3, 14.4, 16.4, 16.5

• 18. Final checkpoint — Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

Notes

• Tasks marked with * are optional and can be skipped for faster MVP
• Each task references specific requirements for traceability
• Checkpoints ensure incremental validation after each major phase
• Property tests validate the 14 universal correctness properties from the design document
• Unit tests validate specific examples, edge cases, and integration points
• The design uses Python throughout — no language selection needed
• Migration number is 034 (existing migrations go up to 033)
• All new dataclass fields use optional defaults for backward compatibility
• Feature flag probabilistic_scoring_enabled gates every behavioral change