Celes Renata
4e010bc048
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
Implement full probabilistic signal processing pipeline gated behind probabilistic_scoring_enabled feature flag in risk_configs: - Bayesian log-likelihood accumulator with Beta posterior and entropy - Regime detector (trend-following, panic, mean-reversion, uncertainty) - Source accuracy tracker with per-source historical prediction accuracy - Sigmoid confidence gate replacing binary gate - Information gain surprise weighting for rare events - Adaptive recency decay with event-specific half-lives - Regime multiplier replacing market context multiplier - Weighted disagreement entropy for contradiction detection - Multiplicative macro exposure with conditional integration - Graph-distance attenuated competitive signal propagation - Exponentially weighted momentum with volatility scaling - Expected value recommendation gate All changes backward-compatible: flag=false preserves exact current behavior. New outputs stored in existing JSONB columns (no schema changes except source_accuracy table via migration 034). Tests: 26 property-based tests (14 correctness properties), 99 unit tests, 1789 total tests passing with zero regressions.
350 lines
21 KiB
Markdown
350 lines
21 KiB
Markdown
# Implementation Plan: Signal Math Upgrade
|
||
|
||
## Overview
|
||
|
||
Upgrade the Stonks Oracle signal processing pipeline from deterministic heuristic formulas to a probabilistic, regime-aware, and adaptive mathematical framework. Implementation proceeds in layers: foundations (config, schemas, new modules), then each pipeline stage (scoring → trend assembly → macro → competitive → projection → recommendation), then integration wiring, and finally testing. All changes are gated behind the `probabilistic_scoring_enabled` feature flag.
|
||
|
||
## Tasks
|
||
|
||
- [ ] 1. Foundation: Configuration and schema extensions
|
||
- [x] 1.1 Extend `ScoringConfig` with probabilistic parameters in `services/aggregation/scoring.py`
|
||
- Add `probabilistic: bool = False` toggle field
|
||
- Add sigmoid gate parameters: `sigmoid_steepness`, `sigmoid_midpoint`
|
||
- Add information gain parameters: `info_gain_lambda`, `info_gain_max`, `default_base_rate`
|
||
- Add adaptive decay parameters: `adaptive_decay_impact_scale`, `adaptive_decay_surprise_scale`, `adaptive_decay_market_scale`
|
||
- Add regime multiplier parameters: `regime_return_weight`, `regime_volume_weight`, `regime_multiplier_max`
|
||
- All new fields must have defaults matching the design document values
|
||
- _Requirements: 2.5, 3.1, 5.1, 6.3, 16.1_
|
||
|
||
- [x] 1.2 Extend `SignalWeight` and `WeightedSignal` dataclasses in `services/aggregation/scoring.py`
|
||
- Add optional fields to `SignalWeight`: `sigmoid_gate`, `info_gain_factor`, `source_accuracy_factor`, `regime_multiplier`
|
||
- Add optional fields to `WeightedSignal`: `info_gain_factor`, `source_accuracy_factor`, `adaptive_half_life`
|
||
- All new fields must have defaults (None or 1.0) for backward compatibility
|
||
- _Requirements: 16.1, 2.5, 3.3, 4.2_
|
||
|
||
- [x] 1.3 Extend `TrendSummary` Pydantic model in `services/shared/schemas.py`
|
||
- Add optional fields: `p_bull`, `alpha`, `beta_param`, `bayesian_confidence`, `entropy`, `regime`, `pipeline_mode`
|
||
- `pipeline_mode` defaults to `"heuristic"`; all others default to `None`
|
||
- _Requirements: 16.1, 1.6, 9.6_
|
||
|
||
- [x] 1.4 Extend `Recommendation` model in `services/shared/schemas.py` (or `services/recommendation/eligibility.py`)
|
||
- Add optional fields: `expected_value`, `p_bull`, `pipeline_mode`
|
||
- `pipeline_mode` defaults to `"heuristic"`; all others default to `None`
|
||
- _Requirements: 16.1, 14.5_
|
||
|
||
- [x] 1.5 Add `probabilistic_scoring_enabled` feature flag support in `services/shared/config.py`
|
||
- Read `probabilistic_scoring_enabled` from `risk_configs.config` JSONB
|
||
- Default to `False` when key is missing, value is invalid, or DB is unreachable
|
||
- Propagate flag through `AggregationConfig` dataclass
|
||
- Log which pipeline mode is active at cycle start
|
||
- _Requirements: 16.3, 16.4, 16.5, 16.6, 16.7_
|
||
|
||
- [x] 1.6 Create database migration `infra/migrations/034_source_accuracy.sql`
|
||
- Create `source_accuracy` table with columns: `id UUID PRIMARY KEY DEFAULT gen_random_uuid()`, `source_id VARCHAR(200) NOT NULL`, `accuracy_ratio FLOAT NOT NULL DEFAULT 0.5`, `sample_count INTEGER NOT NULL DEFAULT 0`, `last_updated TIMESTAMPTZ`, `created_at TIMESTAMPTZ`
|
||
- Add `UNIQUE(source_id)` constraint and `idx_source_accuracy_source` index
|
||
- _Requirements: 4.5_
|
||
|
||
- [x] 2. Checkpoint — Verify foundation compiles and existing tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [ ] 3. New module: Bayesian Accumulator (`services/aggregation/bayesian.py`)
|
||
- [x] 3.1 Implement `BayesianPosterior` dataclass and `compute_bayesian_posterior` function
|
||
- Create frozen dataclass with fields: `p_bull`, `alpha`, `beta`, `log_likelihood`, `bayesian_confidence`, `entropy`, `signal_count`
|
||
- Define `PRIOR` class-level constant for uninformative prior (p_bull=0.5, α=1.0, β=1.0, C=0.0, H=1.0)
|
||
- Implement log-likelihood accumulation: `L_t = Σ(w_i · s_i)` using `weight.combined * sentiment_value`
|
||
- Compute `P_bull = σ(L_t)` via sigmoid function
|
||
- Compute Beta posterior: `α = 1 + W_bull`, `β = 1 + W_bear` from positive/negative weight sums
|
||
- Compute Bayesian confidence: `C = 1 - 4αβ/(α+β)²`
|
||
- Compute Shannon entropy via `compute_entropy`
|
||
- Return `PRIOR` for empty signal lists
|
||
- Skip signals with NaN weight or sentiment
|
||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6_
|
||
|
||
- [x] 3.2 Implement `compute_entropy` function
|
||
- Shannon entropy: `H = -p·log₂(p) - (1-p)·log₂(1-p)`
|
||
- Return 0.0 for p ≤ 0 or p ≥ 1 (edge cases)
|
||
- Return value in [0, 1] with maximum 1.0 at p=0.5
|
||
- _Requirements: 9.1, 9.7_
|
||
|
||
- [x] 3.3 Write property test for sigmoid gate monotonicity
|
||
- **Property 1: Sigmoid Gate Monotonicity**
|
||
- **Validates: Requirements 2.6, 17.1**
|
||
|
||
- [x] 3.4 Write property test for Beta posterior evidence accumulation
|
||
- **Property 2: Beta Posterior Evidence Accumulation**
|
||
- **Validates: Requirements 1.3, 17.2**
|
||
|
||
- [x] 3.5 Write property test for Bayesian confidence symmetry and divergence
|
||
- **Property 3: Bayesian Confidence Symmetry and Divergence**
|
||
- **Validates: Requirements 1.4, 17.3**
|
||
|
||
- [x] 3.6 Write property test for Bayesian posterior round-trip consistency
|
||
- **Property 4: Bayesian Posterior Round-Trip Consistency**
|
||
- **Validates: Requirements 1.7, 17.7**
|
||
|
||
- [x] 3.7 Write property test for Shannon entropy range and maximum
|
||
- **Property 8: Shannon Entropy Range and Maximum**
|
||
- **Validates: Requirements 9.7**
|
||
|
||
- [x] 3.8 Write property test for Bayesian confidence monotonic with agreeing signals
|
||
- **Property 13: Bayesian Confidence Monotonic with Agreeing Signals**
|
||
- **Validates: Requirements 8.6**
|
||
|
||
- [ ] 4. New module: Regime Detector (`services/aggregation/regime.py`)
|
||
- [x] 4.1 Implement `MarketRegime` enum, `RegimeClassification` and `RegimeConfig` dataclasses
|
||
- `MarketRegime`: `TREND_FOLLOWING`, `PANIC`, `MEAN_REVERSION`, `UNCERTAINTY`
|
||
- `RegimeClassification`: `regime`, `trend_indicator`, `volatility_ratio`, `bullish_threshold`, `bearish_threshold`, `contradiction_penalty_multiplier`
|
||
- `RegimeConfig`: all configurable parameters with defaults from design
|
||
- _Requirements: 7.3_
|
||
|
||
- [x] 4.2 Implement `compute_ema` and `classify_regime` functions
|
||
- `compute_ema`: exponential moving average over last N values
|
||
- `classify_regime`: compute trend indicator `R = sign(EMA_20 - EMA_100)` and volatility ratio `V_r = σ_20 / σ_100`
|
||
- Classification rules: trend-following (R≠0 AND V_r<1.2), panic (V_r>1.5), mean-reversion (R=0 AND V_r<1.0), uncertainty (all other)
|
||
- Adjust thresholds per regime: panic→±0.10, mean-reversion→±0.20, trend-following→±0.15, uncertainty→±0.15 with contradiction multiplier 0.6
|
||
- Default to uncertainty when data is insufficient (<100 days) or σ values are zero
|
||
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.9_
|
||
|
||
- [ ] 5. New module: Source Accuracy Tracker (`services/aggregation/source_accuracy.py`)
|
||
- [x] 5.1 Implement `SourceAccuracy` dataclass and database functions
|
||
- `SourceAccuracy` dataclass with `source_id`, `accuracy_ratio`, `sample_count`, `last_updated`
|
||
- `accuracy_factor` property: return 1.0 when sample_count < 10, else `0.5 + accuracy_ratio`
|
||
- `fetch_source_accuracy`: batch fetch from `source_accuracy` table via asyncpg
|
||
- `update_source_accuracy`: update accuracy metrics from realized price outcomes
|
||
- Handle DB unreachable: return neutral factor 1.0 for all sources
|
||
- Clamp corrupted accuracy_ratio to [0.0, 1.0]
|
||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
|
||
|
||
- [x] 6. Checkpoint — Verify new modules compile and unit tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [ ] 7. Signal Scorer upgrades (`services/aggregation/scoring.py`)
|
||
- [x] 7.1 Implement sigmoid confidence gate
|
||
- Add `sigmoid_gate(x, steepness, midpoint)` function: `σ(k·(x - midpoint))`
|
||
- When `probabilistic=True`, replace binary gate with sigmoid gate in `compute_signal_weight`
|
||
- When `probabilistic=False`, preserve existing binary gate behavior
|
||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5_
|
||
|
||
- [x] 7.2 Implement information gain surprise weighting
|
||
- Add `EVENT_TYPE_BASE_RATES` constant dict and `DEFAULT_BASE_RATE = 0.1`
|
||
- Add `compute_info_gain(event_type, lambda_param, max_gain, default_base_rate)` function: `r = 1 + λ·(-log₂ P(event_type))`, clamped to max 3.0
|
||
- Integrate as multiplicative factor in combined weight when `probabilistic=True`
|
||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_
|
||
|
||
- [x] 7.3 Implement adaptive recency decay
|
||
- Add `compute_adaptive_half_life(base_half_life, impact_score, info_gain_factor, market_multiplier, config)` function
|
||
- Compute `β_impact`, `β_surprise`, `β_market_reaction` scaling factors per design
|
||
- `τ_i = τ_base · (1 + β_impact) · (1 + β_surprise) · (1 + β_market_reaction)`
|
||
- When `probabilistic=True`, use adaptive half-life in `recency_weight`; otherwise use fixed
|
||
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7_
|
||
|
||
- [x] 7.4 Implement regime multiplier replacing market context multiplier
|
||
- Add `compute_regime_multiplier(returns, volumes, config)` function
|
||
- Compute z-scores for return and volume, then `M_regime = 1 + 0.15·|z_r| + 0.10·|z_v|`
|
||
- Clamp to [1.0, 2.5]; default to 1.0 when data unavailable or σ=0
|
||
- When `probabilistic=True`, use `M_regime` instead of `M_context` in combined weight
|
||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
|
||
|
||
- [x] 7.5 Integrate source accuracy factor into `compute_signal_weight`
|
||
- Accept optional `source_accuracy_factor` parameter
|
||
- When `probabilistic=True`, multiply into combined weight formula
|
||
- When `probabilistic=False`, ignore (factor = 1.0)
|
||
- _Requirements: 4.2, 4.3_
|
||
|
||
- [x] 7.6 Update `compute_signal_weight` to branch on `probabilistic` flag
|
||
- When `probabilistic=True`: use sigmoid gate × recency (adaptive) × credibility × (1 + novelty) × info_gain × source_accuracy × regime_multiplier
|
||
- When `probabilistic=False`: preserve exact current formula (binary gate × recency × credibility × (1 + novelty) × market_context)
|
||
- Populate all new optional fields on `SignalWeight` and `WeightedSignal`
|
||
- _Requirements: 16.4, 16.5_
|
||
|
||
- [x] 7.7 Write property test for information gain monotonicity
|
||
- **Property 6: Information Gain Monotonicity**
|
||
- **Validates: Requirements 3.5**
|
||
|
||
- [x] 7.8 Write property test for adaptive decay lower bound
|
||
- **Property 5: Adaptive Decay Lower Bound**
|
||
- **Validates: Requirements 5.7, 17.4**
|
||
|
||
- [ ] 8. Contradiction upgrade (`services/aggregation/contradiction.py`)
|
||
- [x] 8.1 Implement weighted disagreement entropy contradiction
|
||
- Compute `f_pos = W_positive / (W_positive + W_negative)` and `f_neg = 1 - f_pos`
|
||
- Compute `H_contradiction = -f_pos·log₂(f_pos) - f_neg·log₂(f_neg)`
|
||
- Weight by evidence mass: `contradiction_score = H_contradiction · min(1.0, (W_pos + W_neg) / W_threshold)`
|
||
- Return 0.0 when only one direction exists
|
||
- Preserve existing `ContradictionResult` interface
|
||
- When `probabilistic=False`, preserve existing minority/majority ratio behavior
|
||
- _Requirements: 15.1, 15.2, 15.3, 15.4, 15.5, 15.6, 15.7_
|
||
|
||
- [x] 8.2 Write property test for contradiction entropy monotonicity
|
||
- **Property 9: Contradiction Entropy Monotonicity**
|
||
- **Validates: Requirements 15.7**
|
||
|
||
- [ ] 9. Trend Assembly upgrades (`services/aggregation/worker.py`)
|
||
- [x] 9.1 Integrate Bayesian posterior into trend assembly
|
||
- When `probabilistic=True`, call `compute_bayesian_posterior` on merged signals
|
||
- Use Bayesian confidence formula for trend confidence: `0.5 × C_bayesian + 0.25 × F_count + 0.25 × C_avg_credibility - P_contradiction`
|
||
- Use entropy-based direction: H>0.9→mixed, P_bull>0.65→bullish, P_bull<0.35→bearish, else neutral
|
||
- Apply regime-adjusted thresholds from `RegimeClassification`
|
||
- Populate new `TrendSummary` fields: `p_bull`, `alpha`, `beta_param`, `bayesian_confidence`, `entropy`, `regime`, `pipeline_mode`
|
||
- Store probabilistic outputs in `market_context` JSONB under `"probabilistic"` key
|
||
- When `probabilistic=False`, preserve exact current heuristic behavior
|
||
- _Requirements: 1.1, 1.2, 8.1, 8.2, 8.3, 8.4, 8.5, 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 7.8, 16.4, 16.5_
|
||
|
||
- [x] 9.2 Wire regime detection into the aggregation cycle
|
||
- Call `classify_regime` with closing prices and returns for each ticker
|
||
- Pass `RegimeClassification` to trend assembly for threshold adjustment
|
||
- Default to uncertainty regime when market data is unavailable
|
||
- Persist regime classification in JSONB for auditability
|
||
- _Requirements: 7.1, 7.2, 7.3, 7.8, 7.9_
|
||
|
||
- [ ] 10. Macro scoring upgrade (`services/aggregation/interpolation.py`)
|
||
- [x] 10.1 Implement multiplicative macro exposure formula
|
||
- When `probabilistic=True`, compute `S_macro = severity · (1 - Π_k(1 - w_k · O_k))` instead of linear weighted sum
|
||
- Preserve overlap weights: w_geo=0.35, w_supply=0.25, w_commodity=0.25, w_sector=0.15
|
||
- Preserve severity mapping and resilience modifier
|
||
- When `probabilistic=False`, preserve exact current linear formula
|
||
- _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6_
|
||
|
||
- [x] 10.2 Implement conditional macro signal integration
|
||
- When `probabilistic=True` and both company and macro signals exist, apply macro as multiplicative modifier: `S_adjusted = S_company · clamp(1 + M_macro · sign_alignment, 0.5, 1.5)`
|
||
- When only macro signals exist, fall back to additive behavior with weight 0.3
|
||
- When only company signals exist, use modifier = 1.0
|
||
- Log macro modifier value per ticker
|
||
- When `probabilistic=False`, preserve current additive merge behavior
|
||
- _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5_
|
||
|
||
- [x] 10.3 Write property test for multiplicative macro exposure monotonicity
|
||
- **Property 7: Multiplicative Macro Exposure Monotonicity**
|
||
- **Validates: Requirements 10.7, 17.5**
|
||
|
||
- [ ] 11. Competitive signal upgrade (`services/aggregation/signal_propagation.py`)
|
||
- [x] 11.1 Implement graph-distance attenuation for competitive signals
|
||
- When `probabilistic=True`, compute `S_transfer = S_source · ρ_historical · e^(-d_network)` instead of flat transfer
|
||
- Compute graph distance as shortest path in competitor relationship graph (cap at 3)
|
||
- Use 90-day rolling Pearson correlation for `ρ_historical`; default to 0.3 (same-sector) or 0.1 (cross-sector) when insufficient data (<30 days)
|
||
- Preserve existing relationship strength threshold (R ≥ 0.2) as pre-filter
|
||
- When `probabilistic=False`, preserve exact current flat transfer behavior
|
||
- _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 12.7_
|
||
|
||
- [x] 11.2 Write property test for competitive signal distance attenuation
|
||
- **Property 11: Competitive Signal Distance Attenuation**
|
||
- **Validates: Requirements 12.7**
|
||
|
||
- [ ] 12. Projection upgrade (`services/aggregation/projection.py`)
|
||
- [x] 12.1 Implement exponentially weighted momentum
|
||
- When `probabilistic=True`, compute `M_t = Σ_{k=0}^{K-1} λ^k · ΔS_{t-k}` with λ=0.7, K up to 10
|
||
- Normalize by geometric series sum to produce value in [-1, 1]
|
||
- Fall back to current heuristic when fewer than 2 historical cycles available
|
||
- Compute volatility-scaled momentum: `M_adj = M_t / max(σ_20, 0.01)`, clamped to [-2.0, 2.0]
|
||
- When `probabilistic=False`, preserve exact current simple momentum behavior
|
||
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6_
|
||
|
||
- [x] 12.2 Write property test for exponentially weighted momentum direction
|
||
- **Property 10: Exponentially Weighted Momentum Direction**
|
||
- **Validates: Requirements 13.6, 17.6**
|
||
|
||
- [ ] 13. Recommendation upgrade (`services/recommendation/eligibility.py`)
|
||
- [x] 13.1 Implement expected value recommendation gate
|
||
- When `probabilistic=True`, compute `EV = P_bull · R_up - P_bear · R_down`
|
||
- Estimate `R_up = strength · σ_20 · √(horizon_days)` and `R_down = (1 - strength) · σ_20 · √(horizon_days)`
|
||
- When EV > threshold (default 0.005), allow recommendation through existing gates
|
||
- When EV ≤ threshold, force recommendation to informational mode
|
||
- Persist EV in `risk_checks` JSONB of `recommendation_evaluations`
|
||
- Populate `expected_value`, `p_bull`, `pipeline_mode` on Recommendation model
|
||
- Preserve all existing eligibility gates as additional requirements
|
||
- When `probabilistic=False`, skip EV gate entirely
|
||
- _Requirements: 14.1, 14.2, 14.3, 14.4, 14.5, 14.6_
|
||
|
||
- [x] 13.2 Write property test for expected value directional consistency
|
||
- **Property 12: Expected Value Directional Consistency**
|
||
- **Validates: Requirements 17.8**
|
||
|
||
- [x] 14. Checkpoint — Verify all pipeline stages compile and existing tests still pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [ ] 15. Integration wiring and feature flag plumbing
|
||
- [x] 15.1 Wire feature flag through the aggregation worker entry point
|
||
- Read `probabilistic_scoring_enabled` from `risk_configs` at cycle start in `services/aggregation/worker.py`
|
||
- Pass flag to `ScoringConfig`, trend assembly, contradiction, macro, competitive, and projection stages
|
||
- Log pipeline mode at cycle start
|
||
- Ensure flag is read once per cycle (mid-cycle changes take effect next cycle)
|
||
- _Requirements: 16.3, 16.6, 16.7_
|
||
|
||
- [x] 15.2 Wire source accuracy fetch into the scoring pipeline
|
||
- At cycle start, batch-fetch source accuracy for all source IDs in the current signal set
|
||
- Pass `source_accuracy_factor` to `compute_signal_weight` for each signal
|
||
- Handle DB errors gracefully (default to 1.0)
|
||
- _Requirements: 4.1, 4.2, 4.3_
|
||
|
||
- [x] 15.3 Wire regime detection into the aggregation cycle
|
||
- Fetch closing prices and returns for each ticker from market data
|
||
- Call `classify_regime` and pass result to trend assembly and scoring stages
|
||
- Handle missing market data (default to uncertainty regime)
|
||
- _Requirements: 7.1, 7.8, 7.9_
|
||
|
||
- [x] 15.4 Store probabilistic outputs in existing JSONB columns
|
||
- Store Bayesian fields in `trend_windows.market_context` JSONB under `"probabilistic"` key
|
||
- Store EV fields in `recommendation_evaluations.risk_checks` JSONB
|
||
- Store regime classification in trend window JSONB
|
||
- _Requirements: 16.2_
|
||
|
||
- [ ] 16. Numerical stability and edge case hardening
|
||
- [x] 16.1 Add input validation and edge case guards across all new functions
|
||
- Guard `log₂(0)` in entropy and information gain computations
|
||
- Floor `max(σ_20, 0.01)` for momentum volatility scaling
|
||
- Default to uncertainty regime when σ values are zero
|
||
- Return `M_regime = 1.0` when z-score σ = 0
|
||
- Skip signals with NaN weight or sentiment
|
||
- Clamp all outputs to documented ranges
|
||
- _Requirements: 17.9, 6.4_
|
||
|
||
- [x] 16.2 Write property test for numerical stability across all formulas
|
||
- **Property 14: Numerical Stability Across All Formulas**
|
||
- **Validates: Requirements 17.9, 6.4**
|
||
|
||
- [ ] 17. Unit tests for all new and modified modules
|
||
- [x] 17.1 Write unit tests for Bayesian accumulator (`tests/test_bayesian.py`)
|
||
- Test uninformative prior (empty signals → P_bull=0.5, α=1, β=1, C=0)
|
||
- Test specific sigmoid gate values (x=0.5→0.5, x=0.2→<0.05, x=0.8→>0.95)
|
||
- Test entropy direction mapping (H>0.9→mixed, P_bull>0.65→bullish, etc.)
|
||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_
|
||
|
||
- [x] 17.2 Write unit tests for regime detector (`tests/test_regime.py`)
|
||
- Test specific (R, V_r) → expected regime classification
|
||
- Test threshold adjustments per regime (panic→0.10, mean_reversion→0.20)
|
||
- Test insufficient data fallback to uncertainty
|
||
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.9_
|
||
|
||
- [x] 17.3 Write unit tests for source accuracy tracker (`tests/test_source_accuracy.py`)
|
||
- Test accuracy_factor property: sample_count < 10 → 1.0, else 0.5 + ratio
|
||
- Test corrupted data clamping
|
||
- _Requirements: 4.1, 4.2, 4.3_
|
||
|
||
- [x] 17.4 Write unit tests for signal scoring upgrades (`tests/test_signal_math_unit.py`)
|
||
- Test info gain clamp (very rare event → factor ≤ 3.0)
|
||
- Test default base rate (unknown event type → 0.1)
|
||
- Test adaptive decay edge cases (all zeros → τ_base, all max → 6×τ_base)
|
||
- Test zero overlap → zero macro impact
|
||
- Test max overlap → ≈severity×0.724
|
||
- Test macro fallback behaviors (only macro → additive, only company → no modifier)
|
||
- Test graph distance cutoff (d>3 → no propagation)
|
||
- Test momentum fallback (<2 cycles → heuristic)
|
||
- Test EV threshold behavior (EV>0.005→proceed, EV≤0.005→informational)
|
||
- Test feature flag behaviors (flag=false→heuristic, flag=true→probabilistic)
|
||
- _Requirements: 3.1, 3.4, 5.5, 5.6, 10.3, 10.4, 11.3, 13.3, 14.3, 14.4, 16.4, 16.5_
|
||
|
||
- [x] 18. Final checkpoint — Ensure all tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
## Notes
|
||
|
||
- Tasks marked with `*` are optional and can be skipped for faster MVP
|
||
- Each task references specific requirements for traceability
|
||
- Checkpoints ensure incremental validation after each major phase
|
||
- Property tests validate the 14 universal correctness properties from the design document
|
||
- Unit tests validate specific examples, edge cases, and integration points
|
||
- The design uses Python throughout — no language selection needed
|
||
- Migration number is 034 (existing migrations go up to 033)
|
||
- All new dataclass fields use optional defaults for backward compatibility
|
||
- Feature flag `probabilistic_scoring_enabled` gates every behavioral change
|