Files
stonks-oracle/.kiro/specs/dual-pipeline-signal-engine/requirements.md
T
Celes Renata f468e30af0
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
feat: implement dual-pipeline signal engine service
New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.

Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
  gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading

Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant

Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)
2026-05-02 07:32:26 +00:00

32 KiB

Requirements Document — Dual-Pipeline Signal Engine

Introduction

The Stonks Oracle platform currently operates a single aggregation pipeline that can run in either heuristic or probabilistic mode (toggled via probabilistic_scoring_enabled). This feature replaces the single-pipeline toggle with a dual-pipeline architecture where both pipelines run concurrently per evaluation tick, produce independent verdicts (BUY/WATCH/SKIP), and emit a structured output contract for downstream consumers (trading engine, delta analysis, dashboards).

The dual-pipeline engine introduces:

  • Pipeline A (Heuristic): Deterministic scoring using the existing S_total = S_company + S_macro + S_competitive formula with signal weighting, producing a confidence-gated verdict.
  • Pipeline B (Probabilistic): Bayesian inference using the existing bayesian.py infrastructure with regime-based priors, likelihood ratios, entropy gating, and expected value calculation.
  • Hard Filter Engine: Pre-pipeline filters that short-circuit both pipelines before evaluation.
  • Multi-Timeframe Engine: Signal evaluation across M30, H1, H4, D, W, M timeframes with weighted confluence scoring.
  • Exit Engine: Position-level exit management (stop hit, targets, trailing ATR-based).
  • Delta Analyzer: Compares heuristic vs probabilistic verdicts to generate training signals for future model tuning.
  • Output Formatter: Structured SignalOutput contract consumed by the trading engine and delta analysis.

The design must address the signal independence assumption in the Bayesian pipeline — correlated signals (MA+RSI, Fib+Elliott) require correlation penalty or signal clustering into categories (momentum, structure, volatility, fundamentals) to prevent likelihood ratio stacking inflation.

Glossary

  • Signal_Engine: The top-level orchestrator in services/signal_engine/ that coordinates input normalization, hard filters, both pipelines, delta analysis, and output formatting per evaluation tick.
  • Heuristic_Pipeline: Pipeline A — deterministic scoring that computes S_total = S_company + S_macro + S_competitive with signal weighting and produces a confidence-gated BUY/WATCH/SKIP verdict.
  • Probabilistic_Pipeline: Pipeline B — Bayesian inference pipeline that computes posterior probability via log-likelihood accumulation with regime-based priors, entropy gating, and expected value calculation.
  • Input_Normalizer: The component that ingests multi-timeframe OHLCV data, fundamentals, macro context, and open positions into a unified NormalizedInput structure consumed by both pipelines.
  • Signal_Library: The collection of technical signal evaluators (Fibonacci retracement, MA stack, RSI, Cup & Handle, Elliott Wave) that produce scored signals per timeframe.
  • Multi_Timeframe_Engine: The component that evaluates signals across six timeframes (M30, H1, H4, D, W, M) and computes weighted confluence scores.
  • Hard_Filter_Engine: The pre-pipeline filter stage that evaluates macro bias, valuation score, and earnings proximity to short-circuit evaluation before either pipeline runs.
  • Exit_Engine: The position management component that evaluates stop hits, take-profit targets, and trailing ATR-based stops for open positions.
  • Delta_Analyzer: The component that compares heuristic and probabilistic verdicts, tracks agreement rates, measures confidence deltas, and records disagreement reasons as training signals.
  • Output_Formatter: The component that assembles the structured SignalOutput contract from both pipeline results, delta analysis, and optional trade plan.
  • SignalOutput: The structured output contract containing ticker, timestamp, price, heuristic verdict/confidence/S_total, probabilistic verdict/P_up/entropy/EV_R, delta analysis, and optional trade plan.
  • Verdict: A pipeline decision of BUY, WATCH, or SKIP with associated confidence and reasoning.
  • Confluence: The condition where a signal triggers across multiple timeframes; requires activation on at least 2 timeframes including at least one of D, W, or M.
  • Entropy_Gate: Shannon entropy threshold used in the probabilistic pipeline to detect high-uncertainty states and force SKIP verdicts.
  • EV_R: Expected value per unit of risk, computed as P_up · E[win_R] - (1 - P_up) · 1.0, used as a quality gate in the probabilistic pipeline.
  • Signal_Cluster: A grouping of correlated signals (momentum, structure, volatility, fundamentals) used to prevent likelihood ratio stacking inflation in the Bayesian pipeline.
  • Likelihood_Ratio: The ratio P(signal|up) / P(signal|down) used in Bayesian updating, where P(sig|up) = h·s + (1-h)·(1-s)·0.5.
  • Regime_Prior: The initial probability assigned based on market regime classification: bull=0.58, range=0.50, bear=0.42.
  • OHLCV: Open, High, Low, Close, Volume — standard market data bar format.
  • ATR: Average True Range — a volatility measure used for trailing stop calculations.
  • Fibonacci_Retracement: A technical analysis tool computing price levels as L(r) = SH - r·(SH - SL) where SH is swing high, SL is swing low, and r is a retracement ratio (0.236, 0.382, 0.5, 0.618, 0.786).

Requirements

Requirement 1: Input Normalization

User Story: As a signal engine operator, I want all market data, fundamentals, macro context, and open positions normalized into a single input structure, so that both pipelines consume identical inputs per evaluation tick.

Acceptance Criteria

  1. WHEN an evaluation tick is triggered for a ticker, THE Input_Normalizer SHALL construct a NormalizedInput containing multi-timeframe OHLCV bars (M30, H1, H4, D, W, M), fundamental metrics (valuation_score, earnings_proximity_days), macro context (macro_bias as float in [-1.0, 1.0]), and open position state (entry_price, current_price, stop_loss, targets).
  2. THE Input_Normalizer SHALL source OHLCV data from the existing market data tables, fundamental metrics from the existing company and trend data, and macro context from the existing macro_impact_records and global_events tables.
  3. IF any required data source is unavailable or returns an error, THEN THE Input_Normalizer SHALL populate the corresponding field with a sentinel value (None for optional fields, empty list for OHLCV bars) and log a warning identifying the missing source.
  4. THE Input_Normalizer SHALL validate that all OHLCV bars have monotonically increasing timestamps within each timeframe series.
  5. THE Input_Normalizer SHALL produce identical NormalizedInput instances for both pipelines within the same evaluation tick (shared reference, no independent fetches).

Requirement 2: Signal Library — Technical Signal Evaluation

User Story: As a quantitative analyst, I want a library of technical signal evaluators that produce scored signals per timeframe, so that both pipelines can consume standardized signal assessments.

Acceptance Criteria

  1. THE Signal_Library SHALL implement Fibonacci retracement signal evaluation using the formula L(r) = SH - r·(SH - SL) for retracement ratios [0.236, 0.382, 0.5, 0.618, 0.786], where SH is the swing high and SL is the swing low within the evaluation window.
  2. THE Signal_Library SHALL implement moving average stack evaluation that detects bullish alignment (MA_10 > MA_20 > MA_50 > MA_200) and bearish alignment (MA_10 < MA_20 < MA_50 < MA_200), producing a signal strength proportional to the degree of alignment.
  3. THE Signal_Library SHALL implement RSI evaluation using the standard 14-period RSI formula, producing overbought signals (RSI > 70) and oversold signals (RSI < 30) with strength scaled by distance from the threshold.
  4. THE Signal_Library SHALL implement Cup & Handle pattern detection that identifies the cup formation (U-shaped price recovery) and handle (small consolidation), producing a signal with confidence proportional to pattern completeness.
  5. THE Signal_Library SHALL implement Elliott Wave detection that identifies impulse waves (5-wave structure) and corrective waves (3-wave structure), producing a signal with the current wave position and projected direction.
  6. WHEN a signal evaluator receives insufficient data for its calculation (fewer bars than the required lookback period), THE Signal_Library SHALL return a null signal with a reason code indicating insufficient data rather than producing a partial evaluation.
  7. FOR ALL signal evaluators, THE Signal_Library SHALL produce output conforming to a common SignalResult structure containing: signal_type, timeframe, strength (float in [0.0, 1.0]), direction (bullish/bearish/neutral), confidence (float in [0.0, 1.0]), and metadata specific to the signal type.

Requirement 3: Multi-Timeframe Confluence Engine

User Story: As a quantitative analyst, I want signals evaluated across multiple timeframes with weighted confluence scoring, so that the engine prioritizes signals confirmed across longer timeframes.

Acceptance Criteria

  1. THE Multi_Timeframe_Engine SHALL evaluate each signal type across six timeframes with the following weights: M30=0.03, H1=0.07, H4=0.15, D=0.30, W=0.30, M=0.15.
  2. THE Multi_Timeframe_Engine SHALL compute a weighted confluence score as C_confluence = Σ(w_tf · s_tf) where w_tf is the timeframe weight and s_tf is the signal strength on that timeframe (0.0 if the signal did not trigger).
  3. WHEN a signal triggers on fewer than 2 timeframes, THE Multi_Timeframe_Engine SHALL discard the signal from further pipeline processing (minimum confluence threshold).
  4. WHEN a signal triggers on 2 or more timeframes but none of D, W, or M are included, THE Multi_Timeframe_Engine SHALL discard the signal from further pipeline processing (higher-timeframe anchor requirement).
  5. THE Multi_Timeframe_Engine SHALL pass the confluence-filtered signals and their weighted scores to both the Heuristic_Pipeline and Probabilistic_Pipeline.
  6. FOR ALL signal sets where a signal triggers on more timeframes with higher weights, THE Multi_Timeframe_Engine SHALL produce a higher confluence score (monotonicity with respect to timeframe activation count and weight).

Requirement 4: Hard Filter Engine — Pre-Pipeline Gating

User Story: As a risk manager, I want hard filters that short-circuit both pipelines before evaluation, so that clearly unfavorable conditions produce immediate SKIP verdicts without wasting computation.

Acceptance Criteria

  1. WHEN the macro_bias value from the NormalizedInput equals -1.0, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "macro_bias_negative".
  2. WHEN the valuation_score from the NormalizedInput is below 0.3, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "valuation_below_threshold".
  3. WHEN the earnings_proximity_days from the NormalizedInput is 5 or fewer, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "earnings_block".
  4. WHEN multiple hard filters trigger simultaneously, THE Hard_Filter_Engine SHALL record all triggered filter reasons in the SKIP verdict (not just the first).
  5. WHEN no hard filters trigger, THE Hard_Filter_Engine SHALL pass the NormalizedInput through to both pipelines without modification.
  6. THE Hard_Filter_Engine SHALL execute before either pipeline begins evaluation, and both pipelines SHALL receive the same filter decision.

Requirement 5: Heuristic Pipeline — Deterministic Scoring and Verdict

User Story: As a quantitative analyst, I want the heuristic pipeline to produce a deterministic BUY/WATCH/SKIP verdict based on composite scoring of company, macro, and competitive signals, so that the system maintains a transparent, auditable scoring path.

Acceptance Criteria

  1. THE Heuristic_Pipeline SHALL compute a total score S_total = S_company + S_macro + S_competitive using the existing three-layer signal aggregation with the current WeightedSignal abstraction.
  2. THE Heuristic_Pipeline SHALL compute signal weights using the formula W_signal = gate · recency · credibility · (1 + novelty) · market_context consistent with the existing compute_signal_weight function in scoring.py.
  3. THE Heuristic_Pipeline SHALL compute a confidence value from the existing trend confidence formula incorporating source count, extraction confidence, signal agreement, and contradiction penalty.
  4. THE Heuristic_Pipeline SHALL produce a BUY verdict WHEN confidence >= 0.70 AND S_total >= 1.2 AND valuation_score >= 0.5 AND macro_bias > 0 AND earnings_proximity_days > 5.
  5. THE Heuristic_Pipeline SHALL produce a WATCH verdict WHEN confidence >= 0.55 AND the BUY conditions are not fully met.
  6. THE Heuristic_Pipeline SHALL produce a SKIP verdict WHEN confidence < 0.55.
  7. THE Heuristic_Pipeline SHALL emit a HeuristicResult containing: verdict (BUY/WATCH/SKIP), confidence (float), S_total (float), S_company (float), S_macro (float), S_competitive (float), signal_weights (list), and reasoning (list of strings explaining the verdict).

Requirement 6: Probabilistic Pipeline — Bayesian Inference and Verdict

User Story: As a quantitative analyst, I want the probabilistic pipeline to produce a Bayesian BUY/WATCH/SKIP verdict using regime-based priors, likelihood ratios, entropy gating, and expected value calculation, so that the system captures uncertainty structure and risk-adjusted expected outcomes.

Acceptance Criteria

  1. THE Probabilistic_Pipeline SHALL initialize the prior probability based on the current market regime classification: bull regime → P_prior = 0.58, range regime → P_prior = 0.50, bear regime → P_prior = 0.42.
  2. THE Probabilistic_Pipeline SHALL compute likelihood ratios for each signal using P(sig|up) = h·s + (1-h)·(1-s)·0.5 and LR = P(sig|up) / P(sig|down), where h is the signal's historical hit rate and s is the signal strength.
  3. THE Probabilistic_Pipeline SHALL update the posterior using log-odds accumulation: logit(P_post) = logit(P_prior) + Σ log(LR_i), converting back to probability via the sigmoid function.
  4. THE Probabilistic_Pipeline SHALL compute Shannon entropy H = -P_up·log₂(P_up) - (1-P_up)·log₂(1-P_up) and apply entropy gating: WHEN H > 0.95, THE Probabilistic_Pipeline SHALL force a SKIP verdict with reason "high_entropy".
  5. THE Probabilistic_Pipeline SHALL compute expected value per unit risk as EV_R = P_up · E[win_R] - (1 - P_up) · 1.0 where E[win_R] is the expected win in risk units derived from signal strength and historical reward-risk ratios.
  6. THE Probabilistic_Pipeline SHALL produce a BUY verdict WHEN P_up >= 0.60 AND entropy <= 0.90 AND EV_R >= 1.5 AND macro_bias > 0 AND valuation_score >= 0.5.
  7. THE Probabilistic_Pipeline SHALL produce a WATCH verdict WHEN P_up >= 0.55 AND entropy <= 0.95 AND the BUY conditions are not fully met.
  8. THE Probabilistic_Pipeline SHALL produce a SKIP verdict in all other cases.
  9. THE Probabilistic_Pipeline SHALL emit a ProbabilisticResult containing: verdict (BUY/WATCH/SKIP), P_up (float), entropy (float), EV_R (float), prior (float), posterior (float), likelihood_ratios (list), regime (string), and reasoning (list of strings).

Requirement 7: Signal Correlation Penalty — Preventing LR Stacking Inflation

User Story: As a quantitative analyst, I want correlated signals grouped into clusters with a correlation penalty applied to prevent likelihood ratio stacking inflation, so that the Bayesian pipeline does not overstate confidence from redundant signals.

Acceptance Criteria

  1. THE Probabilistic_Pipeline SHALL classify each signal into one of four clusters: momentum (MA stack, RSI), structure (Fibonacci retracement, Elliott Wave), volatility (ATR-based signals, Bollinger-derived), and fundamentals (valuation, earnings, macro).
  2. WHEN multiple signals within the same cluster produce likelihood ratios in the same direction, THE Probabilistic_Pipeline SHALL apply a within-cluster penalty: only the strongest LR in the cluster contributes at full weight, and subsequent LRs in the same cluster contribute at a decay factor of 0.5^(n-1) where n is the signal's rank within the cluster by LR magnitude.
  3. THE Probabilistic_Pipeline SHALL apply no penalty across different clusters (signals from different clusters are treated as independent).
  4. WHEN a cluster contains only one signal, THE Probabilistic_Pipeline SHALL apply no penalty to that signal.
  5. FOR ALL signal sets, THE Probabilistic_Pipeline SHALL produce a posterior probability that is less than or equal to the posterior computed without the correlation penalty (the penalty only reduces confidence, never inflates it).

Requirement 8: Exit Engine — Position Management

User Story: As a trader, I want the signal engine to evaluate exit conditions for open positions, so that stop hits, take-profit targets, and trailing stops are managed as part of the signal evaluation cycle.

Acceptance Criteria

  1. WHEN the current price of an open position hits or crosses below the stop_loss level, THE Exit_Engine SHALL emit an EXIT_FULL signal for that position with reason "stop_hit".
  2. WHEN the current price of an open position hits or crosses above the first take-profit target (target_1), THE Exit_Engine SHALL emit an EXIT_HALF signal for that position with reason "target_1_hit".
  3. WHEN the current price of an open position hits or crosses above the second take-profit target (target_2), THE Exit_Engine SHALL emit an EXIT_FULL signal for that position with reason "target_2_hit".
  4. WHEN a partial exit has been executed (EXIT_HALF), THE Exit_Engine SHALL activate a trailing stop at current_price - ATR · trailing_multiplier and update the trailing stop upward as the price advances (the trailing stop moves up but does not move down).
  5. WHEN the trailing stop is active and the current price crosses below the trailing stop level, THE Exit_Engine SHALL emit an EXIT_FULL signal for the remaining position with reason "trailing_stop_hit".
  6. THE Exit_Engine SHALL evaluate exit conditions before the signal pipelines run for new entry signals, so that exit signals take priority over new entry signals for the same ticker.
  7. THE Exit_Engine SHALL emit exit signals as part of the SignalOutput contract with the position identifier, exit type (EXIT_HALF/EXIT_FULL), and reason.

Requirement 9: Delta Analyzer — Pipeline Agreement Tracking

User Story: As a model developer, I want the delta analyzer to compare heuristic and probabilistic verdicts and record disagreement details, so that I can generate training signals for future model tuning.

Acceptance Criteria

  1. WHEN both pipelines produce verdicts for the same ticker and tick, THE Delta_Analyzer SHALL compute an agreement flag (true if both verdicts are identical, false otherwise).
  2. THE Delta_Analyzer SHALL compute a confidence delta as |heuristic_confidence - probabilistic_P_up| representing the magnitude of disagreement between the two pipelines.
  3. WHEN the pipelines disagree on verdict, THE Delta_Analyzer SHALL record the disagreement reason by identifying which conditions differed (e.g., "heuristic_confidence_below_threshold", "probabilistic_entropy_too_high", "EV_R_below_threshold").
  4. THE Delta_Analyzer SHALL track a rolling agreement rate over the last 100 evaluations per ticker, stored in Redis for dashboard consumption.
  5. THE Delta_Analyzer SHALL emit a DeltaResult containing: agreement (bool), confidence_delta (float), heuristic_verdict (string), probabilistic_verdict (string), disagreement_reasons (list of strings), and rolling_agreement_rate (float).
  6. WHEN the rolling agreement rate drops below 0.50 for a ticker, THE Delta_Analyzer SHALL log a warning indicating persistent pipeline disagreement for operator review.

Requirement 10: Output Formatter — Structured SignalOutput Contract

User Story: As a downstream system consumer, I want the signal engine to emit a structured SignalOutput contract, so that the trading engine, delta analysis dashboard, and audit systems can consume a consistent output format.

Acceptance Criteria

  1. THE Output_Formatter SHALL produce a SignalOutput containing: ticker (string), timestamp (datetime), price (float), heuristic section (verdict, confidence, S_total), probabilistic section (verdict, P_up, entropy, EV_R), delta section (agreement, confidence_delta, disagreement_reasons), and optional trade_plan section.
  2. WHEN the heuristic pipeline produces a BUY verdict, THE Output_Formatter SHALL populate the trade_plan section with entry_price, stop_loss, target_1, target_2, and position_size derived from the heuristic confidence and existing position sizing logic.
  3. WHEN the probabilistic pipeline produces a BUY verdict but the heuristic pipeline does not, THE Output_Formatter SHALL populate the trade_plan section with a "probabilistic_only" flag and reduced position sizing (50% of standard).
  4. WHEN both pipelines produce a BUY verdict, THE Output_Formatter SHALL populate the trade_plan section with full position sizing and a "dual_confirmed" flag.
  5. THE Output_Formatter SHALL serialize the SignalOutput as a Pydantic model with JSON serialization support for Redis queue publishing and database persistence.
  6. FOR ALL valid pipeline results, THE Output_Formatter SHALL produce a SignalOutput that round-trips through JSON serialization and deserialization without data loss (parse(format(output)) produces an equivalent object).

Requirement 11: Dual Pipeline Orchestration

User Story: As a signal engine operator, I want both pipelines to run concurrently per evaluation tick sharing the same inputs, so that the system produces independent verdicts without redundant data fetching.

Acceptance Criteria

  1. WHEN an evaluation tick is triggered, THE Signal_Engine SHALL execute the Input_Normalizer once, then pass the resulting NormalizedInput to the Hard_Filter_Engine, then (if not filtered) execute both the Heuristic_Pipeline and Probabilistic_Pipeline concurrently using asyncio.gather.
  2. THE Signal_Engine SHALL enforce that both pipelines receive identical NormalizedInput references (no independent data fetches that could produce different snapshots).
  3. WHEN either pipeline raises an exception during evaluation, THE Signal_Engine SHALL catch the exception, log the error with full traceback, and produce a SKIP verdict for the failed pipeline with reason "pipeline_error" while allowing the other pipeline to complete normally.
  4. THE Signal_Engine SHALL measure and log the wall-clock execution time of each pipeline per tick for performance monitoring.
  5. THE Signal_Engine SHALL publish the assembled SignalOutput to the existing Redis queue (stonks:queue:trading_decisions) for consumption by the trading engine.
  6. THE Signal_Engine SHALL persist each SignalOutput to a database table for historical analysis and audit.

Requirement 12: Integration with Existing Trading Engine

User Story: As a platform operator, I want the dual-pipeline signal engine to integrate with the existing trading engine, so that the trading engine can consume SignalOutput verdicts and make execution decisions.

Acceptance Criteria

  1. THE Signal_Engine SHALL publish SignalOutput to the existing stonks:queue:trading_decisions Redis queue in a format compatible with the existing TradingEngine.evaluate_recommendation interface.
  2. THE Signal_Engine SHALL map the SignalOutput trade_plan to the existing Recommendation schema fields (action, confidence, position_sizing) so that the trading engine can process dual-pipeline outputs without modification to its core evaluation logic.
  3. WHEN the SignalOutput has a "dual_confirmed" flag, THE Signal_Engine SHALL set the recommendation confidence to the maximum of heuristic_confidence and probabilistic_P_up.
  4. WHEN the SignalOutput has a "probabilistic_only" flag, THE Signal_Engine SHALL set the recommendation confidence to probabilistic_P_up · 0.8 (20% confidence haircut for single-pipeline confirmation).
  5. WHEN neither pipeline produces a BUY verdict, THE Signal_Engine SHALL not publish a trading recommendation to the queue (WATCH and SKIP verdicts are persisted for analysis but not forwarded to the trading engine).

Requirement 13: Configuration and Feature Flags

User Story: As a platform operator, I want the dual-pipeline engine configurable via the existing risk_configs table and environment variables, so that I can tune thresholds, enable/disable individual pipelines, and adjust timeframe weights without code changes.

Acceptance Criteria

  1. THE Signal_Engine SHALL support a dual_pipeline_enabled feature flag in risk_configs that toggles the entire dual-pipeline engine on or off, defaulting to false for safe rollout.
  2. THE Signal_Engine SHALL support independent enable/disable flags for each pipeline: heuristic_pipeline_enabled and probabilistic_pipeline_enabled, both defaulting to true when the dual-pipeline engine is enabled.
  3. THE Signal_Engine SHALL support configurable timeframe weights via a timeframe_weights JSON object in risk_configs, defaulting to {"M30": 0.03, "H1": 0.07, "H4": 0.15, "D": 0.30, "W": 0.30, "M": 0.15}.
  4. THE Signal_Engine SHALL support configurable hard filter thresholds: hard_filter_valuation_min (default 0.3), hard_filter_earnings_days (default 5), and hard_filter_macro_bias_skip (default -1.0).
  5. THE Signal_Engine SHALL support configurable verdict thresholds for both pipelines via risk_configs JSON, including heuristic confidence thresholds (BUY: 0.70, WATCH: 0.55) and probabilistic thresholds (P_up: 0.60, entropy: 0.90, EV_R: 1.5).
  6. IF the dual_pipeline_enabled flag fails to read from the database, THEN THE Signal_Engine SHALL default to disabled (fail-safe behavior) and log a warning.
  7. THE Signal_Engine SHALL log the active configuration at startup and on each configuration change for auditability.

Requirement 14: Regime-Based Prior Engine

User Story: As a quantitative analyst, I want the probabilistic pipeline's prior probability to adapt based on the current market regime, so that the Bayesian inference starts from a regime-appropriate baseline rather than a fixed 0.50.

Acceptance Criteria

  1. THE Probabilistic_Pipeline SHALL use the existing classify_regime function from services/aggregation/regime.py to determine the current market regime for each ticker.
  2. THE Probabilistic_Pipeline SHALL map regime classifications to prior probabilities: trend_following with positive trend_indicator → 0.58 (bull), trend_following with negative trend_indicator → 0.42 (bear), mean_reversion → 0.50 (range), panic → 0.42 (bear), uncertainty → 0.50 (range).
  3. THE Probabilistic_Pipeline SHALL convert the regime prior to log-odds before accumulating likelihood ratios: logit(P_prior) = log(P_prior / (1 - P_prior)).
  4. WHEN market data is insufficient for regime classification (fewer than 100 days of price history), THE Probabilistic_Pipeline SHALL use the uncertainty prior of 0.50.
  5. THE Probabilistic_Pipeline SHALL record the regime classification and prior probability in the ProbabilisticResult for auditability.

Requirement 15: Database Schema for Signal Engine Output

User Story: As a platform operator, I want signal engine outputs persisted to a dedicated database table, so that historical evaluations are available for analysis, backtesting, and audit.

Acceptance Criteria

  1. THE Signal_Engine SHALL persist each SignalOutput to a signal_engine_outputs table with columns for: id (UUID primary key), ticker (text), evaluated_at (timestamptz), price (numeric), heuristic_verdict (text), heuristic_confidence (numeric), heuristic_s_total (numeric), probabilistic_verdict (text), probabilistic_p_up (numeric), probabilistic_entropy (numeric), probabilistic_ev_r (numeric), delta_agreement (boolean), delta_confidence_delta (numeric), delta_reasons (JSONB), trade_plan (JSONB), full_output (JSONB), created_at (timestamptz).
  2. THE Signal_Engine SHALL create an index on (ticker, evaluated_at) for efficient time-range queries per ticker.
  3. THE Signal_Engine SHALL create an index on evaluated_at for efficient global time-range queries.
  4. WHEN persisting fails due to a database error, THE Signal_Engine SHALL log the error and continue processing (persistence failure does not block signal emission to the trading queue).

Requirement 16: Backward Compatibility and Migration Path

User Story: As a platform operator, I want the dual-pipeline engine to coexist with the existing single-pipeline aggregation, so that the rollout is incremental and reversible.

Acceptance Criteria

  1. WHEN dual_pipeline_enabled is false, THE Signal_Engine SHALL not run, and the existing aggregation pipeline SHALL continue to operate unchanged.
  2. WHEN dual_pipeline_enabled is true, THE Signal_Engine SHALL run alongside the existing aggregation pipeline, with the trading engine consuming SignalOutput from the dual-pipeline engine instead of Recommendation from the existing recommendation worker.
  3. THE Signal_Engine SHALL reuse the existing WeightedSignal, BayesianPosterior, RegimeClassification, and TrendSummary data structures from services/aggregation/ rather than duplicating them.
  4. THE Signal_Engine SHALL reuse the existing compute_signal_weight, compute_bayesian_posterior, and classify_regime functions rather than reimplementing the underlying math.
  5. THE Signal_Engine SHALL add the new signal_engine_outputs table via a new database migration without modifying existing tables.
  6. THE Signal_Engine SHALL support running in "shadow mode" where both the existing pipeline and the dual-pipeline engine run, but only the existing pipeline's output is forwarded to the trading engine (dual-pipeline output is persisted for comparison only).

Requirement 17: Property-Based Testing for Dual-Pipeline Correctness

User Story: As a developer, I want comprehensive property-based tests validating the mathematical correctness and structural invariants of the dual-pipeline engine, so that edge cases and numerical stability issues are caught before deployment.

Acceptance Criteria

  1. THE test suite SHALL include property-based tests for the Fibonacci retracement formula verifying that L(r) = SH - r·(SH - SL) produces values in [SL, SH] for all r in [0, 1] and all SH > SL > 0.
  2. THE test suite SHALL include property-based tests for the Bayesian log-odds update verifying that logit(P_post) = logit(P_prior) + Σ log(LR_i) round-trips correctly: converting P_prior to logit, adding log-LRs, and converting back via sigmoid produces a valid probability in (0, 1).
  3. THE test suite SHALL include property-based tests for the entropy gate verifying that Shannon entropy is maximized at P_up = 0.5 and equals 0.0 at P_up = 0.0 or P_up = 1.0, and is symmetric around 0.5.
  4. THE test suite SHALL include property-based tests for the signal correlation penalty verifying that the penalized posterior is always less than or equal to the unpenalized posterior for any signal set with correlated signals.
  5. THE test suite SHALL include property-based tests for the multi-timeframe confluence score verifying monotonicity: activating a signal on an additional timeframe with non-zero weight always increases or maintains the confluence score.
  6. THE test suite SHALL include property-based tests for the SignalOutput contract verifying round-trip serialization: SignalOutput.model_validate_json(output.model_dump_json()) produces an equivalent object for all valid outputs.
  7. THE test suite SHALL include property-based tests for the hard filter engine verifying that macro_bias = -1.0 always produces SKIP, valuation_score < 0.3 always produces SKIP, and earnings_proximity_days <= 5 always produces SKIP, regardless of all other input values.
  8. THE test suite SHALL include property-based tests for the EV_R calculation verifying that EV_R = P_up · E[win_R] - (1 - P_up) · 1.0 is monotonically increasing with P_up for fixed E[win_R] > 0.