stonks-oracle/.kiro/specs/dual-pipeline-signal-engine/tasks.md

# Implementation Plan: Dual-Pipeline Signal Engine

## Overview

Implement the dual-pipeline signal engine as a new service at `services/signal_engine/` that runs as an independent Kubernetes deployment. The engine evaluates both a heuristic (deterministic scoring) and probabilistic (Bayesian inference) pipeline concurrently per ticker per evaluation tick, producing independent BUY/WATCH/SKIP verdicts. Implementation proceeds incrementally: infrastructure first, then core models, signal library, pipelines, orchestration, integration, and deployment.

## Tasks

- [x] 1. Project scaffolding, configuration, and data models
  - [x] 1.1 Create service directory structure and `__init__.py` files
    - Create `services/signal_engine/` with all subdirectories per the design module structure
    - Create `services/signal_engine/__init__.py`, `services/signal_engine/signals/__init__.py`
    - _Requirements: 11.1, 13.1_

  - [x] 1.2 Implement `models.py` — all Pydantic data models
    - Define `OHLCVBar`, `NormalizedInput`, `OpenPositionState`, `SignalResult`, `SignalDirection`
    - Define `ConfluenceSignal`, `Verdict`, `HeuristicResult`, `LikelihoodRatio`, `ProbabilisticResult`
    - Define `DeltaResult`, `ExitSignal`, `ExitType`, `TradePlan`, `SignalOutput`
    - All models must use Pydantic `BaseModel` with proper field constraints (`ge`, `le`)
    - _Requirements: 1.1, 2.7, 5.7, 6.9, 9.5, 10.1, 10.5_

  - [x] 1.3 Implement `config.py` — `SignalEngineConfig` and sub-configs
    - Define `SignalEngineConfig` dataclass with all fields from the design
    - Define `HardFilterConfig`, `HeuristicConfig`, `ProbabilisticConfig`, `ExitConfig` as derived sub-configs
    - Implement `load_config()` that reads from `risk_configs` table + environment variables
    - Default `dual_pipeline_enabled` to `False` (fail-safe)
    - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7_

  - [x] 1.4 Add `QUEUE_SIGNAL_ENGINE` to `services/shared/redis_keys.py`
    - Add `QUEUE_SIGNAL_ENGINE = "signal_engine"` constant
    - _Requirements: 11.1_

  - [x] 1.5 Write property test for `SignalOutput` round-trip serialization
    - **Requirement 17.6: SignalOutput round-trip serialization**
    - Generate arbitrary valid `SignalOutput` instances with Hypothesis
    - Verify `SignalOutput.model_validate_json(output.model_dump_json())` produces equivalent object
    - File: `tests/test_pbt_signal_engine_models.py`
    - _Requirements: 10.5, 17.6_

- [x] 2. Input Normalizer and Hard Filter Engine
  - [x] 2.1 Implement `normalizer.py` — Input Normalizer
    - Implement `normalize_input(pool, ticker, config) -> NormalizedInput`
    - Fetch OHLCV bars from `market_data_bars` for M30, H1, H4, D, W, M timeframes
    - Fetch fundamental metrics (valuation_score, earnings_proximity_days) from company/trend data
    - Fetch macro context (macro_bias) from `macro_impact_records` and `global_events`
    - Fetch open position state from trading engine portfolio tables
    - Populate sentinel values (`None`, empty list) for unavailable data with logged warnings
    - Validate monotonically increasing timestamps within each timeframe series
    - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_

  - [x] 2.2 Implement `hard_filter.py` — Hard Filter Engine
    - Implement `evaluate_hard_filters(normalized, config) -> HardFilterResult`
    - Check `macro_bias == -1.0` → SKIP with reason "macro_bias_negative"
    - Check `valuation_score < 0.3` → SKIP with reason "valuation_below_threshold"
    - Check `earnings_proximity_days <= 5` → SKIP with reason "earnings_block"
    - Record all triggered filter reasons (not just first)
    - Return `HardFilterResult` with `filtered: bool` and `reasons: list[str]`
    - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_

  - [x] 2.3 Write property tests for hard filter engine
    - **Requirement 17.7: Hard filter determinism**
    - Generate arbitrary `NormalizedInput` with `macro_bias = -1.0` → always SKIP
    - Generate arbitrary `NormalizedInput` with `valuation_score < 0.3` → always SKIP
    - Generate arbitrary `NormalizedInput` with `earnings_proximity_days <= 5` → always SKIP
    - Verify these hold regardless of all other input values
    - File: `tests/test_pbt_signal_engine_hard_filter.py`
    - _Requirements: 4.1, 4.2, 4.3, 17.7_

- [x] 3. Checkpoint — Ensure all tests pass
  - Ensure all tests pass, ask the user if questions arise.

- [x] 4. Signal Library — Technical Signal Evaluators
  - [x] 4.1 Implement `signals/base.py` — SignalEvaluator protocol
    - Define `SignalEvaluator` protocol with `evaluate(bars, timeframe) -> SignalResult | None`
    - Define common helper functions for swing high/low detection, lookback validation
    - _Requirements: 2.6, 2.7_

  - [x] 4.2 Implement `signals/fibonacci.py` — Fibonacci retracement evaluator
    - Implement `L(r) = SH - r·(SH - SL)` for ratios [0.236, 0.382, 0.5, 0.618, 0.786]
    - Detect swing high and swing low within the evaluation window
    - Produce signal strength based on proximity of current price to retracement levels
    - Return `None` with reason code when insufficient data
    - _Requirements: 2.1, 2.6, 2.7_

  - [x] 4.3 Write property test for Fibonacci retracement formula
    - **Requirement 17.1: Fibonacci retracement bounds**
    - For all `r` in [0, 1] and all `SH > SL > 0`, verify `L(r)` is in [SL, SH]
    - File: `tests/test_pbt_signal_engine_fibonacci.py`
    - _Requirements: 2.1, 17.1_

  - [x] 4.4 Implement `signals/ma_stack.py` — Moving average stack evaluator
    - Detect bullish alignment (MA_10 > MA_20 > MA_50 > MA_200)
    - Detect bearish alignment (MA_10 < MA_20 < MA_50 < MA_200)
    - Produce signal strength proportional to degree of alignment
    - Return `None` when insufficient bars for MA_200 calculation
    - _Requirements: 2.2, 2.6, 2.7_

  - [x] 4.5 Implement `signals/rsi.py` — RSI evaluator
    - Implement standard 14-period RSI formula
    - Produce overbought signals (RSI > 70) and oversold signals (RSI < 30)
    - Scale strength by distance from threshold
    - Return `None` when fewer than 14 bars available
    - _Requirements: 2.3, 2.6, 2.7_

  - [x] 4.6 Implement `signals/cup_handle.py` — Cup & Handle pattern detector
    - Identify cup formation (U-shaped price recovery) and handle (small consolidation)
    - Produce signal with confidence proportional to pattern completeness
    - Return `None` when insufficient data or no pattern detected
    - _Requirements: 2.4, 2.6, 2.7_

  - [x] 4.7 Implement `signals/elliott_wave.py` — Elliott Wave detector
    - Identify impulse waves (5-wave structure) and corrective waves (3-wave structure)
    - Produce signal with current wave position and projected direction
    - Return `None` when insufficient data or ambiguous wave count
    - _Requirements: 2.5, 2.6, 2.7_

- [x] 5. Multi-Timeframe Confluence Engine
  - [x] 5.1 Implement `confluence.py` — Multi-Timeframe Engine
    - Implement `compute_confluence(signal_results, weights) -> list[ConfluenceSignal]`
    - Compute weighted confluence score: `C_confluence = Σ(w_tf · s_tf)`
    - Apply minimum confluence threshold: discard signals triggering on < 2 timeframes
    - Apply higher-timeframe anchor: discard signals without at least one of D, W, or M
    - Return `ConfluenceSignal` objects with active timeframes and per-timeframe strengths
    - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_

  - [x] 5.2 Write property test for confluence score monotonicity
    - **Requirement 17.5: Confluence score monotonicity**
    - Verify that activating a signal on an additional timeframe with non-zero weight always increases or maintains the confluence score
    - File: `tests/test_pbt_signal_engine_confluence.py`
    - _Requirements: 3.6, 17.5_

- [x] 6. Checkpoint — Ensure all tests pass
  - Ensure all tests pass, ask the user if questions arise.

- [x] 7. Heuristic Pipeline (Pipeline A)
  - [x] 7.1 Implement `heuristic.py` — Heuristic Pipeline
    - Implement `run_heuristic_pipeline(normalized, confluence_signals, config) -> HeuristicResult`
    - Compute `S_total = S_company + S_macro + S_competitive` using existing `compute_signal_weight()`
    - Compute confidence from source count, extraction confidence, signal agreement, contradiction penalty
    - BUY verdict: confidence >= 0.70 AND S_total >= 1.2 AND valuation_score >= 0.5 AND macro_bias > 0 AND earnings_proximity_days > 5
    - WATCH verdict: confidence >= 0.55 AND BUY conditions not fully met
    - SKIP verdict: confidence < 0.55
    - Emit `HeuristicResult` with all required fields and reasoning
    - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7_

  - [x] 7.2 Write unit tests for heuristic pipeline verdict logic
    - Test BUY threshold conditions
    - Test WATCH threshold conditions
    - Test SKIP conditions
    - Test edge cases at threshold boundaries
    - File: `tests/test_signal_engine_heuristic.py`
    - _Requirements: 5.4, 5.5, 5.6_

- [x] 8. Probabilistic Pipeline (Pipeline B) and Correlation Penalty
  - [x] 8.1 Implement `correlation.py` — Signal cluster classification and penalty
    - Define `SignalCluster` enum: MOMENTUM, STRUCTURE, VOLATILITY, FUNDAMENTALS
    - Implement `classify_signal(signal_type) -> SignalCluster`
    - Implement `apply_correlation_penalty(likelihood_ratios) -> list[LikelihoodRatio]`
    - Within-cluster decay: strongest LR at full weight, subsequent at 0.5^(n-1)
    - No penalty across different clusters
    - Single-signal clusters receive no penalty
    - _Requirements: 7.1, 7.2, 7.3, 7.4_

  - [x] 8.2 Implement `probabilistic.py` — Probabilistic Pipeline
    - Implement `run_probabilistic_pipeline(normalized, confluence_signals, regime, config) -> ProbabilisticResult`
    - Initialize regime-based prior: bull=0.58, range=0.50, bear=0.42
    - Compute likelihood ratios: `P(sig|up) = h·s + (1-h)·(1-s)·0.5`, `LR = P(sig|up) / P(sig|down)`
    - Apply correlation penalty via `apply_correlation_penalty()`
    - Accumulate via log-odds: `logit(P_post) = logit(P_prior) + Σ log(LR_i)`
    - Compute Shannon entropy and apply entropy gating (H > 0.95 → SKIP)
    - Compute `EV_R = P_up · E[win_R] - (1 - P_up) · 1.0`
    - BUY: P_up >= 0.60 AND entropy <= 0.90 AND EV_R >= 1.5 AND macro_bias > 0 AND valuation_score >= 0.5
    - WATCH: P_up >= 0.55 AND entropy <= 0.95 AND BUY conditions not fully met
    - SKIP: all other cases
    - Use existing `classify_regime()` from `services/aggregation/regime.py`
    - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9, 14.1, 14.2, 14.3, 14.4, 14.5_

  - [x] 8.3 Write property test for Bayesian log-odds round-trip
    - **Requirement 17.2: Bayesian log-odds update correctness**
    - Verify `logit(P_post) = logit(P_prior) + Σ log(LR_i)` round-trips correctly
    - Converting P_prior to logit, adding log-LRs, converting back via sigmoid produces valid probability in (0, 1)
    - File: `tests/test_pbt_signal_engine_bayesian.py`
    - _Requirements: 6.3, 17.2_

  - [x] 8.4 Write property test for entropy gate
    - **Requirement 17.3: Entropy gate properties**
    - Verify Shannon entropy is maximized at P_up = 0.5
    - Verify entropy equals 0.0 at P_up = 0.0 or P_up = 1.0
    - Verify entropy is symmetric around 0.5
    - File: `tests/test_pbt_signal_engine_bayesian.py`
    - _Requirements: 6.4, 17.3_

  - [x] 8.5 Write property test for signal correlation penalty
    - **Requirement 17.4: Correlation penalty reduces confidence**
    - Verify penalized posterior is always <= unpenalized posterior for any signal set with correlated signals
    - File: `tests/test_pbt_signal_engine_correlation.py`
    - _Requirements: 7.5, 17.4_

  - [x] 8.6 Write property test for EV_R monotonicity
    - **Requirement 17.8: EV_R monotonically increasing with P_up**
    - Verify `EV_R = P_up · E[win_R] - (1 - P_up) · 1.0` is monotonically increasing with P_up for fixed E[win_R] > 0
    - File: `tests/test_pbt_signal_engine_bayesian.py`
    - _Requirements: 6.5, 17.8_

- [x] 9. Checkpoint — Ensure all tests pass
  - Ensure all tests pass, ask the user if questions arise.

- [x] 10. Exit Engine
  - [x] 10.1 Implement `exit_engine.py` — Exit Engine
    - Implement `evaluate_exits(positions, current_prices, config) -> list[ExitSignal]`
    - Check stop_loss hit → EXIT_FULL with reason "stop_hit"
    - Check target_1 hit → EXIT_HALF with reason "target_1_hit"
    - Check target_2 hit → EXIT_FULL with reason "target_2_hit"
    - Trailing stop: activate after EXIT_HALF at `current_price - ATR · trailing_multiplier`
    - Trailing stop ratchets upward only (never moves down)
    - Trailing stop hit → EXIT_FULL with reason "trailing_stop_hit"
    - _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7_

  - [x] 10.2 Write unit tests for exit engine
    - Test stop_loss trigger
    - Test target_1 partial exit
    - Test target_2 full exit
    - Test trailing stop activation and ratchet behavior
    - File: `tests/test_signal_engine_exit.py`
    - _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5_

- [x] 11. Delta Analyzer and Output Formatter
  - [x] 11.1 Implement `delta.py` — Delta Analyzer
    - Implement `analyze_delta(heuristic, probabilistic, redis, ticker) -> DeltaResult`
    - Compute agreement flag (both verdicts identical)
    - Compute confidence delta: `|heuristic_confidence - probabilistic_P_up|`
    - Record disagreement reasons when verdicts differ
    - Track rolling 100-evaluation agreement rate in Redis
    - Log warning when agreement rate drops below 0.50
    - _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6_

  - [x] 11.2 Implement `formatter.py` — Output Formatter
    - Implement `format_output(ticker, price, heuristic, probabilistic, delta, exit_signals, config) -> SignalOutput`
    - Both BUY → `dual_confirmed`, full position sizing
    - Probabilistic-only BUY → `probabilistic_only`, 50% position sizing
    - Heuristic-only BUY → standard position sizing
    - No BUY → no trade_plan (WATCH/SKIP persisted for analysis)
    - Implement `signal_output_to_recommendation(output) -> Recommendation`
    - Map `SignalOutput` to existing `Recommendation` schema for trading engine compatibility
    - Dual confirmed: confidence = max(heuristic_confidence, probabilistic_P_up)
    - Probabilistic only: confidence = probabilistic_P_up · 0.8 (20% haircut)
    - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 12.1, 12.2, 12.3, 12.4, 12.5_

  - [x] 11.3 Write unit tests for output formatter
    - Test dual_confirmed trade plan generation
    - Test probabilistic_only trade plan with 50% sizing
    - Test heuristic-only trade plan
    - Test no-BUY case (no trade_plan)
    - Test `signal_output_to_recommendation` mapping
    - File: `tests/test_signal_engine_formatter.py`
    - _Requirements: 10.2, 10.3, 10.4, 12.3, 12.4_

- [x] 12. Orchestrator, Persistence, and Main Entry Point
  - [x] 12.1 Implement `persistence.py` — Database persistence
    - Implement `persist_signal_output(pool, output) -> None`
    - Insert into `signal_engine_outputs` table
    - Log and continue on database errors (non-blocking)
    - _Requirements: 15.1, 15.4_

  - [x] 12.2 Implement `worker.py` — Top-level orchestrator
    - Implement `evaluate_tick(pool, redis, ticker, config) -> SignalOutput | None`
    - Step 1: Normalize inputs (single fetch, shared reference)
    - Step 2: Evaluate exit conditions for open positions
    - Step 3: Run hard filters (short-circuit if filtered)
    - Step 4: Evaluate signals across timeframes via Signal Library
    - Step 5: Compute confluence
    - Step 6: Classify regime via existing `classify_regime()`
    - Step 7: Run both pipelines concurrently via `asyncio.gather` with exception handling
    - Step 8: Compute delta analysis
    - Step 9: Format output
    - Step 10: Persist to database and publish to Redis queue
    - Catch pipeline exceptions → SKIP verdict for failed pipeline, other continues
    - Measure and log wall-clock execution time per pipeline
    - _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6_

  - [x] 12.3 Implement `main.py` — Entry point with asyncio event loop
    - Connect to PostgreSQL (asyncpg pool) and Redis (redis.asyncio)
    - Load config from `risk_configs` table
    - Log active configuration at startup
    - Poll `stonks:queue:signal_engine` queue indefinitely
    - Check `dual_pipeline_enabled` flag; if disabled, sleep and retry
    - On config read failure, default to disabled (fail-safe)
    - Support shadow mode (persist but don't forward to trading queue)
    - _Requirements: 13.1, 13.6, 13.7, 16.1, 16.6_

  - [x] 12.4 Write integration tests for worker orchestration
    - Test full tick evaluation with mocked DB/Redis
    - Test pipeline failure isolation (one fails, other completes)
    - Test hard filter short-circuit
    - Test shadow mode behavior
    - File: `tests/test_signal_engine_worker.py`
    - _Requirements: 11.3, 16.6_

- [x] 13. Checkpoint — Ensure all tests pass
  - Ensure all tests pass, ask the user if questions arise.

- [x] 14. Database migration and infrastructure
  - [x] 14.1 Create database migration `infra/migrations/039_signal_engine_outputs.sql`
    - Create `signal_engine_outputs` table per the design schema
    - Create index on `(ticker, evaluated_at)` for per-ticker time-range queries
    - Create index on `evaluated_at` for global time-range queries
    - Create index on `(heuristic_verdict, probabilistic_verdict)` for verdict filtering
    - _Requirements: 15.1, 15.2, 15.3_

  - [x] 14.2 Add signal engine service to Helm chart
    - Add `signalEngine` entry to `infra/helm/stonks-oracle/values.yaml`
    - Configure: replicas=1, command=`python -m services.signal_engine.main`, tier=processing
    - Set resource requests/limits per design (100m/128Mi → 500m/256Mi)
    - Reference existing secrets: `stonks-core-secrets`, `stonks-market-secrets`
    - _Requirements: 11.1, 13.1_

- [x] 15. Trading engine integration and backward compatibility
  - [x] 15.1 Wire signal engine output to trading engine queue
    - Publish `SignalOutput` (mapped to `Recommendation`) to `stonks:queue:trading_decisions`
    - Only publish when at least one pipeline produces BUY verdict
    - WATCH/SKIP verdicts persisted for analysis but not forwarded
    - Ensure trading engine can consume without modification via `signal_output_to_recommendation()`
    - _Requirements: 12.1, 12.2, 12.5, 16.2_

  - [x] 15.2 Ensure backward compatibility with existing pipeline
    - Verify `dual_pipeline_enabled=false` means signal engine does not run
    - Verify existing aggregation pipeline operates unchanged when flag is off
    - Reuse existing `WeightedSignal`, `BayesianPosterior`, `RegimeClassification` (import, don't duplicate)
    - Reuse existing `compute_signal_weight`, `compute_bayesian_posterior`, `classify_regime` functions
    - No modifications to existing tables (new migration only adds new table)
    - _Requirements: 16.1, 16.2, 16.3, 16.4, 16.5_

- [x] 16. Final checkpoint — Ensure all tests pass
  - Ensure all tests pass, ask the user if questions arise.

## Notes

- Tasks marked with `*` are optional and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation between major phases
- Property-based tests use Hypothesis with `@settings(max_examples=100)` per project conventions
- PBT test files are prefixed `test_pbt_*` per project conventions
- The service reuses existing math functions from `services/aggregation/` — no reimplementation
- All configuration is loaded from `risk_configs` table with fail-safe defaults
- Shadow mode allows running alongside existing pipeline without affecting trading decisions