# Implementation Plan: Signal Math Upgrade

## Overview

Upgrade the Stonks Oracle signal processing pipeline from deterministic heuristic formulas to a probabilistic, regime-aware, and adaptive mathematical framework. Implementation proceeds in layers: foundations (config, schemas, new modules), then each pipeline stage (scoring → trend assembly → macro → competitive → projection → recommendation), then integration wiring, and finally testing. All changes are gated behind the `probabilistic_scoring_enabled` feature flag.

## Tasks

- [ ] 1. Foundation: Configuration and schema extensions
  - [x] 1.1 Extend `ScoringConfig` with probabilistic parameters in `services/aggregation/scoring.py`
    - Add `probabilistic: bool = False` toggle field
    - Add sigmoid gate parameters: `sigmoid_steepness`, `sigmoid_midpoint`
    - Add information gain parameters: `info_gain_lambda`, `info_gain_max`, `default_base_rate`
    - Add adaptive decay parameters: `adaptive_decay_impact_scale`, `adaptive_decay_surprise_scale`, `adaptive_decay_market_scale`
    - Add regime multiplier parameters: `regime_return_weight`, `regime_volume_weight`, `regime_multiplier_max`
    - All new fields must have defaults matching the design document values
    - _Requirements: 2.5, 3.1, 5.1, 6.3, 16.1_

  - [x] 1.2 Extend `SignalWeight` and `WeightedSignal` dataclasses in `services/aggregation/scoring.py`
    - Add optional fields to `SignalWeight`: `sigmoid_gate`, `info_gain_factor`, `source_accuracy_factor`, `regime_multiplier`
    - Add optional fields to `WeightedSignal`: `info_gain_factor`, `source_accuracy_factor`, `adaptive_half_life`
    - All new fields must have defaults (None or 1.0) for backward compatibility
    - _Requirements: 16.1, 2.5, 3.3, 4.2_

  - [x] 1.3 Extend `TrendSummary` Pydantic model in `services/shared/schemas.py`
    - Add optional fields: `p_bull`, `alpha`, `beta_param`, `bayesian_confidence`, `entropy`, `regime`, `pipeline_mode`
    - `pipeline_mode` defaults to `"heuristic"`; all others default to `None`
    - _Requirements: 16.1, 1.6, 9.6_

  - [x] 1.4 Extend `Recommendation` model in `services/shared/schemas.py` (or `services/recommendation/eligibility.py`)
    - Add optional fields: `expected_value`, `p_bull`, `pipeline_mode`
    - `pipeline_mode` defaults to `"heuristic"`; all others default to `None`
    - _Requirements: 16.1, 14.5_

  - [x] 1.5 Add `probabilistic_scoring_enabled` feature flag support in `services/shared/config.py`
    - Read `probabilistic_scoring_enabled` from `risk_configs.config` JSONB
    - Default to `False` when key is missing, value is invalid, or DB is unreachable
    - Propagate flag through `AggregationConfig` dataclass
    - Log which pipeline mode is active at cycle start
    - _Requirements: 16.3, 16.4, 16.5, 16.6, 16.7_

  - [x] 1.6 Create database migration `infra/migrations/034_source_accuracy.sql`
    - Create `source_accuracy` table with columns: `id UUID PRIMARY KEY DEFAULT gen_random_uuid()`, `source_id VARCHAR(200) NOT NULL`, `accuracy_ratio FLOAT NOT NULL DEFAULT 0.5`, `sample_count INTEGER NOT NULL DEFAULT 0`, `last_updated TIMESTAMPTZ`, `created_at TIMESTAMPTZ`
    - Add `UNIQUE(source_id)` constraint and `idx_source_accuracy_source` index
    - _Requirements: 4.5_

- [x] 2. Checkpoint — Verify foundation compiles and existing tests pass
  - Ensure all tests pass, ask the user if questions arise.

- [ ] 3. New module: Bayesian Accumulator (`services/aggregation/bayesian.py`)
  - [x] 3.1 Implement `BayesianPosterior` dataclass and `compute_bayesian_posterior` function
    - Create frozen dataclass with fields: `p_bull`, `alpha`, `beta`, `log_likelihood`, `bayesian_confidence`, `entropy`, `signal_count`
    - Define `PRIOR` class-level constant for uninformative prior (p_bull=0.5, α=1.0, β=1.0, C=0.0, H=1.0)
    - Implement log-likelihood accumulation: `L_t = Σ(w_i · s_i)` using `weight.combined * sentiment_value`
    - Compute `P_bull = σ(L_t)` via sigmoid function
    - Compute Beta posterior: `α = 1 + W_bull`, `β = 1 + W_bear` from positive/negative weight sums
    - Compute Bayesian confidence: `C = 1 - 4αβ/(α+β)²`
    - Compute Shannon entropy via `compute_entropy`
    - Return `PRIOR` for empty signal lists
    - Skip signals with NaN weight or sentiment
    - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_

  - [x] 3.2 Implement `compute_entropy` function
    - Shannon entropy: `H = -p·log₂(p) - (1-p)·log₂(1-p)`
    - Return 0.0 for p ≤ 0 or p ≥ 1 (edge cases)
    - Return value in [0, 1] with maximum 1.0 at p=0.5
    - _Requirements: 9.1, 9.7_

  - [x] 3.3 Write property test for sigmoid gate monotonicity
    - **Property 1: Sigmoid Gate Monotonicity**
    - **Validates: Requirements 2.6, 17.1**

  - [x] 3.4 Write property test for Beta posterior evidence accumulation
    - **Property 2: Beta Posterior Evidence Accumulation**
    - **Validates: Requirements 1.3, 17.2**

  - [x] 3.5 Write property test for Bayesian confidence symmetry and divergence
    - **Property 3: Bayesian Confidence Symmetry and Divergence**
    - **Validates: Requirements 1.4, 17.3**

  - [x] 3.6 Write property test for Bayesian posterior round-trip consistency
    - **Property 4: Bayesian Posterior Round-Trip Consistency**
    - **Validates: Requirements 1.7, 17.7**

  - [x] 3.7 Write property test for Shannon entropy range and maximum
    - **Property 8: Shannon Entropy Range and Maximum**
    - **Validates: Requirements 9.7**

  - [x] 3.8 Write property test for Bayesian confidence monotonic with agreeing signals
    - **Property 13: Bayesian Confidence Monotonic with Agreeing Signals**
    - **Validates: Requirements 8.6**

- [ ] 4. New module: Regime Detector (`services/aggregation/regime.py`)
  - [x] 4.1 Implement `MarketRegime` enum, `RegimeClassification` and `RegimeConfig` dataclasses
    - `MarketRegime`: `TREND_FOLLOWING`, `PANIC`, `MEAN_REVERSION`, `UNCERTAINTY`
    - `RegimeClassification`: `regime`, `trend_indicator`, `volatility_ratio`, `bullish_threshold`, `bearish_threshold`, `contradiction_penalty_multiplier`
    - `RegimeConfig`: all configurable parameters with defaults from design
    - _Requirements: 7.3_

  - [x] 4.2 Implement `compute_ema` and `classify_regime` functions
    - `compute_ema`: exponential moving average over last N values
    - `classify_regime`: compute trend indicator `R = sign(EMA_20 - EMA_100)` and volatility ratio `V_r = σ_20 / σ_100`
    - Classification rules: trend-following (R≠0 AND V_r<1.2), panic (V_r>1.5), mean-reversion (R=0 AND V_r<1.0), uncertainty (all other)
    - Adjust thresholds per regime: panic→±0.10, mean-reversion→±0.20, trend-following→±0.15, uncertainty→±0.15 with contradiction multiplier 0.6
    - Default to uncertainty when data is insufficient (<100 days) or σ values are zero
    - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.9_

- [ ] 5. New module: Source Accuracy Tracker (`services/aggregation/source_accuracy.py`)
  - [x] 5.1 Implement `SourceAccuracy` dataclass and database functions
    - `SourceAccuracy` dataclass with `source_id`, `accuracy_ratio`, `sample_count`, `last_updated`
    - `accuracy_factor` property: return 1.0 when sample_count < 10, else `0.5 + accuracy_ratio`
    - `fetch_source_accuracy`: batch fetch from `source_accuracy` table via asyncpg
    - `update_source_accuracy`: update accuracy metrics from realized price outcomes
    - Handle DB unreachable: return neutral factor 1.0 for all sources
    - Clamp corrupted accuracy_ratio to [0.0, 1.0]
    - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_

- [x] 6. Checkpoint — Verify new modules compile and unit tests pass
  - Ensure all tests pass, ask the user if questions arise.

- [ ] 7. Signal Scorer upgrades (`services/aggregation/scoring.py`)
  - [x] 7.1 Implement sigmoid confidence gate
    - Add `sigmoid_gate(x, steepness, midpoint)` function: `σ(k·(x - midpoint))`
    - When `probabilistic=True`, replace binary gate with sigmoid gate in `compute_signal_weight`
    - When `probabilistic=False`, preserve existing binary gate behavior
    - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5_

  - [x] 7.2 Implement information gain surprise weighting
    - Add `EVENT_TYPE_BASE_RATES` constant dict and `DEFAULT_BASE_RATE = 0.1`
    - Add `compute_info_gain(event_type, lambda_param, max_gain, default_base_rate)` function: `r = 1 + λ·(-log₂ P(event_type))`, clamped to max 3.0
    - Integrate as multiplicative factor in combined weight when `probabilistic=True`
    - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_

  - [x] 7.3 Implement adaptive recency decay
    - Add `compute_adaptive_half_life(base_half_life, impact_score, info_gain_factor, market_multiplier, config)` function
    - Compute `β_impact`, `β_surprise`, `β_market_reaction` scaling factors per design
    - `τ_i = τ_base · (1 + β_impact) · (1 + β_surprise) · (1 + β_market_reaction)`
    - When `probabilistic=True`, use adaptive half-life in `recency_weight`; otherwise use fixed
    - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7_

  - [x] 7.4 Implement regime multiplier replacing market context multiplier
    - Add `compute_regime_multiplier(returns, volumes, config)` function
    - Compute z-scores for return and volume, then `M_regime = 1 + 0.15·|z_r| + 0.10·|z_v|`
    - Clamp to [1.0, 2.5]; default to 1.0 when data unavailable or σ=0
    - When `probabilistic=True`, use `M_regime` instead of `M_context` in combined weight
    - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_

  - [x] 7.5 Integrate source accuracy factor into `compute_signal_weight`
    - Accept optional `source_accuracy_factor` parameter
    - When `probabilistic=True`, multiply into combined weight formula
    - When `probabilistic=False`, ignore (factor = 1.0)
    - _Requirements: 4.2, 4.3_

  - [x] 7.6 Update `compute_signal_weight` to branch on `probabilistic` flag
    - When `probabilistic=True`: use sigmoid gate × recency (adaptive) × credibility × (1 + novelty) × info_gain × source_accuracy × regime_multiplier
    - When `probabilistic=False`: preserve exact current formula (binary gate × recency × credibility × (1 + novelty) × market_context)
    - Populate all new optional fields on `SignalWeight` and `WeightedSignal`
    - _Requirements: 16.4, 16.5_

  - [x] 7.7 Write property test for information gain monotonicity
    - **Property 6: Information Gain Monotonicity**
    - **Validates: Requirements 3.5**

  - [x] 7.8 Write property test for adaptive decay lower bound
    - **Property 5: Adaptive Decay Lower Bound**
    - **Validates: Requirements 5.7, 17.4**

- [ ] 8. Contradiction upgrade (`services/aggregation/contradiction.py`)
  - [x] 8.1 Implement weighted disagreement entropy contradiction
    - Compute `f_pos = W_positive / (W_positive + W_negative)` and `f_neg = 1 - f_pos`
    - Compute `H_contradiction = -f_pos·log₂(f_pos) - f_neg·log₂(f_neg)`
    - Weight by evidence mass: `contradiction_score = H_contradiction · min(1.0, (W_pos + W_neg) / W_threshold)`
    - Return 0.0 when only one direction exists
    - Preserve existing `ContradictionResult` interface
    - When `probabilistic=False`, preserve existing minority/majority ratio behavior
    - _Requirements: 15.1, 15.2, 15.3, 15.4, 15.5, 15.6, 15.7_

  - [x] 8.2 Write property test for contradiction entropy monotonicity
    - **Property 9: Contradiction Entropy Monotonicity**
    - **Validates: Requirements 15.7**

- [ ] 9. Trend Assembly upgrades (`services/aggregation/worker.py`)
  - [x] 9.1 Integrate Bayesian posterior into trend assembly
    - When `probabilistic=True`, call `compute_bayesian_posterior` on merged signals
    - Use Bayesian confidence formula for trend confidence: `0.5 × C_bayesian + 0.25 × F_count + 0.25 × C_avg_credibility - P_contradiction`
    - Use entropy-based direction: H>0.9→mixed, P_bull>0.65→bullish, P_bull<0.35→bearish, else neutral
    - Apply regime-adjusted thresholds from `RegimeClassification`
    - Populate new `TrendSummary` fields: `p_bull`, `alpha`, `beta_param`, `bayesian_confidence`, `entropy`, `regime`, `pipeline_mode`
    - Store probabilistic outputs in `market_context` JSONB under `"probabilistic"` key
    - When `probabilistic=False`, preserve exact current heuristic behavior
    - _Requirements: 1.1, 1.2, 8.1, 8.2, 8.3, 8.4, 8.5, 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 7.8, 16.4, 16.5_

  - [x] 9.2 Wire regime detection into the aggregation cycle
    - Call `classify_regime` with closing prices and returns for each ticker
    - Pass `RegimeClassification` to trend assembly for threshold adjustment
    - Default to uncertainty regime when market data is unavailable
    - Persist regime classification in JSONB for auditability
    - _Requirements: 7.1, 7.2, 7.3, 7.8, 7.9_

- [ ] 10. Macro scoring upgrade (`services/aggregation/interpolation.py`)
  - [x] 10.1 Implement multiplicative macro exposure formula
    - When `probabilistic=True`, compute `S_macro = severity · (1 - Π_k(1 - w_k · O_k))` instead of linear weighted sum
    - Preserve overlap weights: w_geo=0.35, w_supply=0.25, w_commodity=0.25, w_sector=0.15
    - Preserve severity mapping and resilience modifier
    - When `probabilistic=False`, preserve exact current linear formula
    - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6_

  - [x] 10.2 Implement conditional macro signal integration
    - When `probabilistic=True` and both company and macro signals exist, apply macro as multiplicative modifier: `S_adjusted = S_company · clamp(1 + M_macro · sign_alignment, 0.5, 1.5)`
    - When only macro signals exist, fall back to additive behavior with weight 0.3
    - When only company signals exist, use modifier = 1.0
    - Log macro modifier value per ticker
    - When `probabilistic=False`, preserve current additive merge behavior
    - _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5_

  - [x] 10.3 Write property test for multiplicative macro exposure monotonicity
    - **Property 7: Multiplicative Macro Exposure Monotonicity**
    - **Validates: Requirements 10.7, 17.5**

- [ ] 11. Competitive signal upgrade (`services/aggregation/signal_propagation.py`)
  - [x] 11.1 Implement graph-distance attenuation for competitive signals
    - When `probabilistic=True`, compute `S_transfer = S_source · ρ_historical · e^(-d_network)` instead of flat transfer
    - Compute graph distance as shortest path in competitor relationship graph (cap at 3)
    - Use 90-day rolling Pearson correlation for `ρ_historical`; default to 0.3 (same-sector) or 0.1 (cross-sector) when insufficient data (<30 days)
    - Preserve existing relationship strength threshold (R ≥ 0.2) as pre-filter
    - When `probabilistic=False`, preserve exact current flat transfer behavior
    - _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 12.7_

  - [x] 11.2 Write property test for competitive signal distance attenuation
    - **Property 11: Competitive Signal Distance Attenuation**
    - **Validates: Requirements 12.7**

- [ ] 12. Projection upgrade (`services/aggregation/projection.py`)
  - [x] 12.1 Implement exponentially weighted momentum
    - When `probabilistic=True`, compute `M_t = Σ_{k=0}^{K-1} λ^k · ΔS_{t-k}` with λ=0.7, K up to 10
    - Normalize by geometric series sum to produce value in [-1, 1]
    - Fall back to current heuristic when fewer than 2 historical cycles available
    - Compute volatility-scaled momentum: `M_adj = M_t / max(σ_20, 0.01)`, clamped to [-2.0, 2.0]
    - When `probabilistic=False`, preserve exact current simple momentum behavior
    - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6_

  - [x] 12.2 Write property test for exponentially weighted momentum direction
    - **Property 10: Exponentially Weighted Momentum Direction**
    - **Validates: Requirements 13.6, 17.6**

- [ ] 13. Recommendation upgrade (`services/recommendation/eligibility.py`)
  - [x] 13.1 Implement expected value recommendation gate
    - When `probabilistic=True`, compute `EV = P_bull · R_up - P_bear · R_down`
    - Estimate `R_up = strength · σ_20 · √(horizon_days)` and `R_down = (1 - strength) · σ_20 · √(horizon_days)`
    - When EV > threshold (default 0.005), allow recommendation through existing gates
    - When EV ≤ threshold, force recommendation to informational mode
    - Persist EV in `risk_checks` JSONB of `recommendation_evaluations`
    - Populate `expected_value`, `p_bull`, `pipeline_mode` on Recommendation model
    - Preserve all existing eligibility gates as additional requirements
    - When `probabilistic=False`, skip EV gate entirely
    - _Requirements: 14.1, 14.2, 14.3, 14.4, 14.5, 14.6_

  - [x] 13.2 Write property test for expected value directional consistency
    - **Property 12: Expected Value Directional Consistency**
    - **Validates: Requirements 17.8**

- [x] 14. Checkpoint — Verify all pipeline stages compile and existing tests still pass
  - Ensure all tests pass, ask the user if questions arise.

- [ ] 15. Integration wiring and feature flag plumbing
  - [x] 15.1 Wire feature flag through the aggregation worker entry point
    - Read `probabilistic_scoring_enabled` from `risk_configs` at cycle start in `services/aggregation/worker.py`
    - Pass flag to `ScoringConfig`, trend assembly, contradiction, macro, competitive, and projection stages
    - Log pipeline mode at cycle start
    - Ensure flag is read once per cycle (mid-cycle changes take effect next cycle)
    - _Requirements: 16.3, 16.6, 16.7_

  - [x] 15.2 Wire source accuracy fetch into the scoring pipeline
    - At cycle start, batch-fetch source accuracy for all source IDs in the current signal set
    - Pass `source_accuracy_factor` to `compute_signal_weight` for each signal
    - Handle DB errors gracefully (default to 1.0)
    - _Requirements: 4.1, 4.2, 4.3_

  - [x] 15.3 Wire regime detection into the aggregation cycle
    - Fetch closing prices and returns for each ticker from market data
    - Call `classify_regime` and pass result to trend assembly and scoring stages
    - Handle missing market data (default to uncertainty regime)
    - _Requirements: 7.1, 7.8, 7.9_

  - [x] 15.4 Store probabilistic outputs in existing JSONB columns
    - Store Bayesian fields in `trend_windows.market_context` JSONB under `"probabilistic"` key
    - Store EV fields in `recommendation_evaluations.risk_checks` JSONB
    - Store regime classification in trend window JSONB
    - _Requirements: 16.2_

- [ ] 16. Numerical stability and edge case hardening
  - [x] 16.1 Add input validation and edge case guards across all new functions
    - Guard `log₂(0)` in entropy and information gain computations
    - Floor `max(σ_20, 0.01)` for momentum volatility scaling
    - Default to uncertainty regime when σ values are zero
    - Return `M_regime = 1.0` when z-score σ = 0
    - Skip signals with NaN weight or sentiment
    - Clamp all outputs to documented ranges
    - _Requirements: 17.9, 6.4_

  - [x] 16.2 Write property test for numerical stability across all formulas
    - **Property 14: Numerical Stability Across All Formulas**
    - **Validates: Requirements 17.9, 6.4**

- [ ] 17. Unit tests for all new and modified modules
  - [x] 17.1 Write unit tests for Bayesian accumulator (`tests/test_bayesian.py`)
    - Test uninformative prior (empty signals → P_bull=0.5, α=1, β=1, C=0)
    - Test specific sigmoid gate values (x=0.5→0.5, x=0.2→<0.05, x=0.8→>0.95)
    - Test entropy direction mapping (H>0.9→mixed, P_bull>0.65→bullish, etc.)
    - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_

  - [x] 17.2 Write unit tests for regime detector (`tests/test_regime.py`)
    - Test specific (R, V_r) → expected regime classification
    - Test threshold adjustments per regime (panic→0.10, mean_reversion→0.20)
    - Test insufficient data fallback to uncertainty
    - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.9_

  - [x] 17.3 Write unit tests for source accuracy tracker (`tests/test_source_accuracy.py`)
    - Test accuracy_factor property: sample_count < 10 → 1.0, else 0.5 + ratio
    - Test corrupted data clamping
    - _Requirements: 4.1, 4.2, 4.3_

  - [x] 17.4 Write unit tests for signal scoring upgrades (`tests/test_signal_math_unit.py`)
    - Test info gain clamp (very rare event → factor ≤ 3.0)
    - Test default base rate (unknown event type → 0.1)
    - Test adaptive decay edge cases (all zeros → τ_base, all max → 6×τ_base)
    - Test zero overlap → zero macro impact
    - Test max overlap → ≈severity×0.724
    - Test macro fallback behaviors (only macro → additive, only company → no modifier)
    - Test graph distance cutoff (d>3 → no propagation)
    - Test momentum fallback (<2 cycles → heuristic)
    - Test EV threshold behavior (EV>0.005→proceed, EV≤0.005→informational)
    - Test feature flag behaviors (flag=false→heuristic, flag=true→probabilistic)
    - _Requirements: 3.1, 3.4, 5.5, 5.6, 10.3, 10.4, 11.3, 13.3, 14.3, 14.4, 16.4, 16.5_

- [x] 18. Final checkpoint — Ensure all tests pass
  - Ensure all tests pass, ask the user if questions arise.

## Notes

- Tasks marked with `*` are optional and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation after each major phase
- Property tests validate the 14 universal correctness properties from the design document
- Unit tests validate specific examples, edge cases, and integration points
- The design uses Python throughout — no language selection needed
- Migration number is 034 (existing migrations go up to 033)
- All new dataclass fields use optional defaults for backward compatibility
- Feature flag `probabilistic_scoring_enabled` gates every behavioral change