stonks-oracle/.kiro/specs/dual-pipeline-signal-engine/tasks.md

Implementation Plan: Dual-Pipeline Signal Engine

Overview

Implement the dual-pipeline signal engine as a new service at services/signal_engine/ that runs as an independent Kubernetes deployment. The engine evaluates both a heuristic (deterministic scoring) and probabilistic (Bayesian inference) pipeline concurrently per ticker per evaluation tick, producing independent BUY/WATCH/SKIP verdicts. Implementation proceeds incrementally: infrastructure first, then core models, signal library, pipelines, orchestration, integration, and deployment.

Tasks

• 1. Project scaffolding, configuration, and data models

  • 1.1 Create service directory structure and __init__.py files

    • Create services/signal_engine/ with all subdirectories per the design module structure
    • Create services/signal_engine/__init__.py, services/signal_engine/signals/__init__.py
    • Requirements: 11.1, 13.1

  • 1.2 Implement models.py — all Pydantic data models

    • Define OHLCVBar, NormalizedInput, OpenPositionState, SignalResult, SignalDirection
    • Define ConfluenceSignal, Verdict, HeuristicResult, LikelihoodRatio, ProbabilisticResult
    • Define DeltaResult, ExitSignal, ExitType, TradePlan, SignalOutput
    • All models must use Pydantic BaseModel with proper field constraints (ge, le)
    • Requirements: 1.1, 2.7, 5.7, 6.9, 9.5, 10.1, 10.5

  • 1.3 Implement config.py — SignalEngineConfig and sub-configs

    • Define SignalEngineConfig dataclass with all fields from the design
    • Define HardFilterConfig, HeuristicConfig, ProbabilisticConfig, ExitConfig as derived sub-configs
    • Implement load_config() that reads from risk_configs table + environment variables
    • Default dual_pipeline_enabled to False (fail-safe)
    • Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7

  • 1.4 Add QUEUE_SIGNAL_ENGINE to services/shared/redis_keys.py

    • Add QUEUE_SIGNAL_ENGINE = "signal_engine" constant
    • Requirements: 11.1

  • 1.5 Write property test for SignalOutput round-trip serialization

    • Requirement 17.6: SignalOutput round-trip serialization
    • Generate arbitrary valid SignalOutput instances with Hypothesis
    • Verify SignalOutput.model_validate_json(output.model_dump_json()) produces equivalent object
    • File: tests/test_pbt_signal_engine_models.py
    • Requirements: 10.5, 17.6

• 2. Input Normalizer and Hard Filter Engine

  • 2.1 Implement normalizer.py — Input Normalizer

    • Implement normalize_input(pool, ticker, config) -> NormalizedInput
    • Fetch OHLCV bars from market_data_bars for M30, H1, H4, D, W, M timeframes
    • Fetch fundamental metrics (valuation_score, earnings_proximity_days) from company/trend data
    • Fetch macro context (macro_bias) from macro_impact_records and global_events
    • Fetch open position state from trading engine portfolio tables
    • Populate sentinel values (None, empty list) for unavailable data with logged warnings
    • Validate monotonically increasing timestamps within each timeframe series
    • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5

  • 2.2 Implement hard_filter.py — Hard Filter Engine

    • Implement evaluate_hard_filters(normalized, config) -> HardFilterResult
    • Check macro_bias == -1.0 → SKIP with reason "macro_bias_negative"
    • Check valuation_score < 0.3 → SKIP with reason "valuation_below_threshold"
    • Check earnings_proximity_days <= 5 → SKIP with reason "earnings_block"
    • Record all triggered filter reasons (not just first)
    • Return HardFilterResult with filtered: bool and reasons: list[str]
    • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6

  • 2.3 Write property tests for hard filter engine

    • Requirement 17.7: Hard filter determinism
    • Generate arbitrary NormalizedInput with macro_bias = -1.0 → always SKIP
    • Generate arbitrary NormalizedInput with valuation_score < 0.3 → always SKIP
    • Generate arbitrary NormalizedInput with earnings_proximity_days <= 5 → always SKIP
    • Verify these hold regardless of all other input values
    • File: tests/test_pbt_signal_engine_hard_filter.py
    • Requirements: 4.1, 4.2, 4.3, 17.7

• 3. Checkpoint — Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

• 4. Signal Library — Technical Signal Evaluators

  • 4.1 Implement signals/base.py — SignalEvaluator protocol

    • Define SignalEvaluator protocol with evaluate(bars, timeframe) -> SignalResult | None
    • Define common helper functions for swing high/low detection, lookback validation
    • Requirements: 2.6, 2.7

  • 4.2 Implement signals/fibonacci.py — Fibonacci retracement evaluator

    • Implement L(r) = SH - r·(SH - SL) for ratios [0.236, 0.382, 0.5, 0.618, 0.786]
    • Detect swing high and swing low within the evaluation window
    • Produce signal strength based on proximity of current price to retracement levels
    • Return None with reason code when insufficient data
    • Requirements: 2.1, 2.6, 2.7

  • 4.3 Write property test for Fibonacci retracement formula

    • Requirement 17.1: Fibonacci retracement bounds
    • For all r in [0, 1] and all SH > SL > 0, verify L(r) is in [SL, SH]
    • File: tests/test_pbt_signal_engine_fibonacci.py
    • Requirements: 2.1, 17.1

  • 4.4 Implement signals/ma_stack.py — Moving average stack evaluator

    • Detect bullish alignment (MA_10 > MA_20 > MA_50 > MA_200)
    • Detect bearish alignment (MA_10 < MA_20 < MA_50 < MA_200)
    • Produce signal strength proportional to degree of alignment
    • Return None when insufficient bars for MA_200 calculation
    • Requirements: 2.2, 2.6, 2.7

  • 4.5 Implement signals/rsi.py — RSI evaluator

    • Implement standard 14-period RSI formula
    • Produce overbought signals (RSI > 70) and oversold signals (RSI < 30)
    • Scale strength by distance from threshold
    • Return None when fewer than 14 bars available
    • Requirements: 2.3, 2.6, 2.7

  • 4.6 Implement signals/cup_handle.py — Cup & Handle pattern detector

    • Identify cup formation (U-shaped price recovery) and handle (small consolidation)
    • Produce signal with confidence proportional to pattern completeness
    • Return None when insufficient data or no pattern detected
    • Requirements: 2.4, 2.6, 2.7

  • 4.7 Implement signals/elliott_wave.py — Elliott Wave detector

    • Identify impulse waves (5-wave structure) and corrective waves (3-wave structure)
    • Produce signal with current wave position and projected direction
    • Return None when insufficient data or ambiguous wave count
    • Requirements: 2.5, 2.6, 2.7

• 5. Multi-Timeframe Confluence Engine

  • 5.1 Implement confluence.py — Multi-Timeframe Engine

    • Implement compute_confluence(signal_results, weights) -> list[ConfluenceSignal]
    • Compute weighted confluence score: C_confluence = Σ(w_tf · s_tf)
    • Apply minimum confluence threshold: discard signals triggering on < 2 timeframes
    • Apply higher-timeframe anchor: discard signals without at least one of D, W, or M
    • Return ConfluenceSignal objects with active timeframes and per-timeframe strengths
    • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6

  • 5.2 Write property test for confluence score monotonicity

    • Requirement 17.5: Confluence score monotonicity
    • Verify that activating a signal on an additional timeframe with non-zero weight always increases or maintains the confluence score
    • File: tests/test_pbt_signal_engine_confluence.py
    • Requirements: 3.6, 17.5

• 6. Checkpoint — Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

• 7. Heuristic Pipeline (Pipeline A)

  • 7.1 Implement heuristic.py — Heuristic Pipeline

    • Implement run_heuristic_pipeline(normalized, confluence_signals, config) -> HeuristicResult
    • Compute S_total = S_company + S_macro + S_competitive using existing compute_signal_weight()
    • Compute confidence from source count, extraction confidence, signal agreement, contradiction penalty
    • BUY verdict: confidence >= 0.70 AND S_total >= 1.2 AND valuation_score >= 0.5 AND macro_bias > 0 AND earnings_proximity_days > 5
    • WATCH verdict: confidence >= 0.55 AND BUY conditions not fully met
    • SKIP verdict: confidence < 0.55
    • Emit HeuristicResult with all required fields and reasoning
    • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7

  • 7.2 Write unit tests for heuristic pipeline verdict logic

    • Test BUY threshold conditions
    • Test WATCH threshold conditions
    • Test SKIP conditions
    • Test edge cases at threshold boundaries
    • File: tests/test_signal_engine_heuristic.py
    • Requirements: 5.4, 5.5, 5.6

• 8. Probabilistic Pipeline (Pipeline B) and Correlation Penalty

  • 8.1 Implement correlation.py — Signal cluster classification and penalty

    • Define SignalCluster enum: MOMENTUM, STRUCTURE, VOLATILITY, FUNDAMENTALS
    • Implement classify_signal(signal_type) -> SignalCluster
    • Implement apply_correlation_penalty(likelihood_ratios) -> list[LikelihoodRatio]
    • Within-cluster decay: strongest LR at full weight, subsequent at 0.5^(n-1)
    • No penalty across different clusters
    • Single-signal clusters receive no penalty
    • Requirements: 7.1, 7.2, 7.3, 7.4

  • 8.2 Implement probabilistic.py — Probabilistic Pipeline

    • Implement run_probabilistic_pipeline(normalized, confluence_signals, regime, config) -> ProbabilisticResult
    • Initialize regime-based prior: bull=0.58, range=0.50, bear=0.42
    • Compute likelihood ratios: P(sig|up) = h·s + (1-h)·(1-s)·0.5, LR = P(sig|up) / P(sig|down)
    • Apply correlation penalty via apply_correlation_penalty()
    • Accumulate via log-odds: logit(P_post) = logit(P_prior) + Σ log(LR_i)
    • Compute Shannon entropy and apply entropy gating (H > 0.95 → SKIP)
    • Compute EV_R = P_up · E[win_R] - (1 - P_up) · 1.0
    • BUY: P_up >= 0.60 AND entropy <= 0.90 AND EV_R >= 1.5 AND macro_bias > 0 AND valuation_score >= 0.5
    • WATCH: P_up >= 0.55 AND entropy <= 0.95 AND BUY conditions not fully met
    • SKIP: all other cases
    • Use existing classify_regime() from services/aggregation/regime.py
    • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9, 14.1, 14.2, 14.3, 14.4, 14.5

  • 8.3 Write property test for Bayesian log-odds round-trip

    • Requirement 17.2: Bayesian log-odds update correctness
    • Verify logit(P_post) = logit(P_prior) + Σ log(LR_i) round-trips correctly
    • Converting P_prior to logit, adding log-LRs, converting back via sigmoid produces valid probability in (0, 1)
    • File: tests/test_pbt_signal_engine_bayesian.py
    • Requirements: 6.3, 17.2

  • 8.4 Write property test for entropy gate

    • Requirement 17.3: Entropy gate properties
    • Verify Shannon entropy is maximized at P_up = 0.5
    • Verify entropy equals 0.0 at P_up = 0.0 or P_up = 1.0
    • Verify entropy is symmetric around 0.5
    • File: tests/test_pbt_signal_engine_bayesian.py
    • Requirements: 6.4, 17.3

  • 8.5 Write property test for signal correlation penalty

    • Requirement 17.4: Correlation penalty reduces confidence
    • Verify penalized posterior is always <= unpenalized posterior for any signal set with correlated signals
    • File: tests/test_pbt_signal_engine_correlation.py
    • Requirements: 7.5, 17.4

  • 8.6 Write property test for EV_R monotonicity

    • Requirement 17.8: EV_R monotonically increasing with P_up
    • Verify EV_R = P_up · E[win_R] - (1 - P_up) · 1.0 is monotonically increasing with P_up for fixed E[win_R] > 0
    • File: tests/test_pbt_signal_engine_bayesian.py
    • Requirements: 6.5, 17.8

• 9. Checkpoint — Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

• 10. Exit Engine

  • 10.1 Implement exit_engine.py — Exit Engine

    • Implement evaluate_exits(positions, current_prices, config) -> list[ExitSignal]
    • Check stop_loss hit → EXIT_FULL with reason "stop_hit"
    • Check target_1 hit → EXIT_HALF with reason "target_1_hit"
    • Check target_2 hit → EXIT_FULL with reason "target_2_hit"
    • Trailing stop: activate after EXIT_HALF at current_price - ATR · trailing_multiplier
    • Trailing stop ratchets upward only (never moves down)
    • Trailing stop hit → EXIT_FULL with reason "trailing_stop_hit"
    • Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7

  • 10.2 Write unit tests for exit engine

    • Test stop_loss trigger
    • Test target_1 partial exit
    • Test target_2 full exit
    • Test trailing stop activation and ratchet behavior
    • File: tests/test_signal_engine_exit.py
    • Requirements: 8.1, 8.2, 8.3, 8.4, 8.5

• 11. Delta Analyzer and Output Formatter

  • 11.1 Implement delta.py — Delta Analyzer

    • Implement analyze_delta(heuristic, probabilistic, redis, ticker) -> DeltaResult
    • Compute agreement flag (both verdicts identical)
    • Compute confidence delta: |heuristic_confidence - probabilistic_P_up|
    • Record disagreement reasons when verdicts differ
    • Track rolling 100-evaluation agreement rate in Redis
    • Log warning when agreement rate drops below 0.50
    • Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6

  • 11.2 Implement formatter.py — Output Formatter

    • Implement format_output(ticker, price, heuristic, probabilistic, delta, exit_signals, config) -> SignalOutput
    • Both BUY → dual_confirmed, full position sizing
    • Probabilistic-only BUY → probabilistic_only, 50% position sizing
    • Heuristic-only BUY → standard position sizing
    • No BUY → no trade_plan (WATCH/SKIP persisted for analysis)
    • Implement signal_output_to_recommendation(output) -> Recommendation
    • Map SignalOutput to existing Recommendation schema for trading engine compatibility
    • Dual confirmed: confidence = max(heuristic_confidence, probabilistic_P_up)
    • Probabilistic only: confidence = probabilistic_P_up · 0.8 (20% haircut)
    • Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 12.1, 12.2, 12.3, 12.4, 12.5

  • 11.3 Write unit tests for output formatter

    • Test dual_confirmed trade plan generation
    • Test probabilistic_only trade plan with 50% sizing
    • Test heuristic-only trade plan
    • Test no-BUY case (no trade_plan)
    • Test signal_output_to_recommendation mapping
    • File: tests/test_signal_engine_formatter.py
    • Requirements: 10.2, 10.3, 10.4, 12.3, 12.4

• 12. Orchestrator, Persistence, and Main Entry Point

  • 12.1 Implement persistence.py — Database persistence

    • Implement persist_signal_output(pool, output) -> None
    • Insert into signal_engine_outputs table
    • Log and continue on database errors (non-blocking)
    • Requirements: 15.1, 15.4

  • 12.2 Implement worker.py — Top-level orchestrator

    • Implement evaluate_tick(pool, redis, ticker, config) -> SignalOutput | None
    • Step 1: Normalize inputs (single fetch, shared reference)
    • Step 2: Evaluate exit conditions for open positions
    • Step 3: Run hard filters (short-circuit if filtered)
    • Step 4: Evaluate signals across timeframes via Signal Library
    • Step 5: Compute confluence
    • Step 6: Classify regime via existing classify_regime()
    • Step 7: Run both pipelines concurrently via asyncio.gather with exception handling
    • Step 8: Compute delta analysis
    • Step 9: Format output
    • Step 10: Persist to database and publish to Redis queue
    • Catch pipeline exceptions → SKIP verdict for failed pipeline, other continues
    • Measure and log wall-clock execution time per pipeline
    • Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6

  • 12.3 Implement main.py — Entry point with asyncio event loop

    • Connect to PostgreSQL (asyncpg pool) and Redis (redis.asyncio)
    • Load config from risk_configs table
    • Log active configuration at startup
    • Poll stonks:queue:signal_engine queue indefinitely
    • Check dual_pipeline_enabled flag; if disabled, sleep and retry
    • On config read failure, default to disabled (fail-safe)
    • Support shadow mode (persist but don't forward to trading queue)
    • Requirements: 13.1, 13.6, 13.7, 16.1, 16.6

  • 12.4 Write integration tests for worker orchestration

    • Test full tick evaluation with mocked DB/Redis
    • Test pipeline failure isolation (one fails, other completes)
    • Test hard filter short-circuit
    • Test shadow mode behavior
    • File: tests/test_signal_engine_worker.py
    • Requirements: 11.3, 16.6

• 13. Checkpoint — Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

• 14. Database migration and infrastructure

  • 14.1 Create database migration infra/migrations/039_signal_engine_outputs.sql

    • Create signal_engine_outputs table per the design schema
    • Create index on (ticker, evaluated_at) for per-ticker time-range queries
    • Create index on evaluated_at for global time-range queries
    • Create index on (heuristic_verdict, probabilistic_verdict) for verdict filtering
    • Requirements: 15.1, 15.2, 15.3

  • 14.2 Add signal engine service to Helm chart

    • Add signalEngine entry to infra/helm/stonks-oracle/values.yaml
    • Configure: replicas=1, command=python -m services.signal_engine.main, tier=processing
    • Set resource requests/limits per design (100m/128Mi → 500m/256Mi)
    • Reference existing secrets: stonks-core-secrets, stonks-market-secrets
    • Requirements: 11.1, 13.1

• 15. Trading engine integration and backward compatibility

  • 15.1 Wire signal engine output to trading engine queue

    • Publish SignalOutput (mapped to Recommendation) to stonks:queue:trading_decisions
    • Only publish when at least one pipeline produces BUY verdict
    • WATCH/SKIP verdicts persisted for analysis but not forwarded
    • Ensure trading engine can consume without modification via signal_output_to_recommendation()
    • Requirements: 12.1, 12.2, 12.5, 16.2

  • 15.2 Ensure backward compatibility with existing pipeline

    • Verify dual_pipeline_enabled=false means signal engine does not run
    • Verify existing aggregation pipeline operates unchanged when flag is off
    • Reuse existing WeightedSignal, BayesianPosterior, RegimeClassification (import, don't duplicate)
    • Reuse existing compute_signal_weight, compute_bayesian_posterior, classify_regime functions
    • No modifications to existing tables (new migration only adds new table)
    • Requirements: 16.1, 16.2, 16.3, 16.4, 16.5

• 16. Final checkpoint — Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

Notes

• Tasks marked with * are optional and can be skipped for faster MVP
• Each task references specific requirements for traceability
• Checkpoints ensure incremental validation between major phases
• Property-based tests use Hypothesis with @settings(max_examples=100) per project conventions
• PBT test files are prefixed test_pbt_* per project conventions
• The service reuses existing math functions from services/aggregation/ — no reimplementation
• All configuration is loaded from risk_configs table with fail-safe defaults
• Shadow mode allows running alongside existing pipeline without affecting trading decisions