New service at services/signal_engine/ implementing concurrent heuristic (deterministic scoring) and probabilistic (Bayesian inference) pipelines that evaluate technical signals across 6 timeframes (M30-M) and produce independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick. Components: - Input Normalizer: multi-source data assembly with sentinel fallbacks - Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave - Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors - Hard Filter Engine: macro_bias, valuation, earnings proximity gating - Heuristic Pipeline: S_total scoring with confidence-gated verdicts - Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy gating, EV_R calculation, and signal correlation penalty - Exit Engine: stop-loss, targets, trailing ATR-based stops - Delta Analyzer: pipeline agreement tracking with rolling Redis metrics - Output Formatter: SignalOutput contract + Recommendation schema mapping - Worker orchestrator: concurrent pipelines with failure isolation - Main entry point: queue polling with fail-safe config loading Infrastructure: - Migration 039: signal_engine_outputs table with 3 indexes - Helm chart: signalEngine service entry (processing tier) - Redis key: QUEUE_SIGNAL_ENGINE constant Tests: 390 tests (unit + property-based) covering all components Config: dual_pipeline_enabled=false by default (safe rollout)
32 KiB
Requirements Document — Dual-Pipeline Signal Engine
Introduction
The Stonks Oracle platform currently operates a single aggregation pipeline that can run in either heuristic or probabilistic mode (toggled via probabilistic_scoring_enabled). This feature replaces the single-pipeline toggle with a dual-pipeline architecture where both pipelines run concurrently per evaluation tick, produce independent verdicts (BUY/WATCH/SKIP), and emit a structured output contract for downstream consumers (trading engine, delta analysis, dashboards).
The dual-pipeline engine introduces:
- Pipeline A (Heuristic): Deterministic scoring using the existing
S_total = S_company + S_macro + S_competitiveformula with signal weighting, producing a confidence-gated verdict. - Pipeline B (Probabilistic): Bayesian inference using the existing
bayesian.pyinfrastructure with regime-based priors, likelihood ratios, entropy gating, and expected value calculation. - Hard Filter Engine: Pre-pipeline filters that short-circuit both pipelines before evaluation.
- Multi-Timeframe Engine: Signal evaluation across M30, H1, H4, D, W, M timeframes with weighted confluence scoring.
- Exit Engine: Position-level exit management (stop hit, targets, trailing ATR-based).
- Delta Analyzer: Compares heuristic vs probabilistic verdicts to generate training signals for future model tuning.
- Output Formatter: Structured
SignalOutputcontract consumed by the trading engine and delta analysis.
The design must address the signal independence assumption in the Bayesian pipeline — correlated signals (MA+RSI, Fib+Elliott) require correlation penalty or signal clustering into categories (momentum, structure, volatility, fundamentals) to prevent likelihood ratio stacking inflation.
Glossary
- Signal_Engine: The top-level orchestrator in
services/signal_engine/that coordinates input normalization, hard filters, both pipelines, delta analysis, and output formatting per evaluation tick. - Heuristic_Pipeline: Pipeline A — deterministic scoring that computes
S_total = S_company + S_macro + S_competitivewith signal weighting and produces a confidence-gated BUY/WATCH/SKIP verdict. - Probabilistic_Pipeline: Pipeline B — Bayesian inference pipeline that computes posterior probability via log-likelihood accumulation with regime-based priors, entropy gating, and expected value calculation.
- Input_Normalizer: The component that ingests multi-timeframe OHLCV data, fundamentals, macro context, and open positions into a unified
NormalizedInputstructure consumed by both pipelines. - Signal_Library: The collection of technical signal evaluators (Fibonacci retracement, MA stack, RSI, Cup & Handle, Elliott Wave) that produce scored signals per timeframe.
- Multi_Timeframe_Engine: The component that evaluates signals across six timeframes (M30, H1, H4, D, W, M) and computes weighted confluence scores.
- Hard_Filter_Engine: The pre-pipeline filter stage that evaluates macro bias, valuation score, and earnings proximity to short-circuit evaluation before either pipeline runs.
- Exit_Engine: The position management component that evaluates stop hits, take-profit targets, and trailing ATR-based stops for open positions.
- Delta_Analyzer: The component that compares heuristic and probabilistic verdicts, tracks agreement rates, measures confidence deltas, and records disagreement reasons as training signals.
- Output_Formatter: The component that assembles the structured
SignalOutputcontract from both pipeline results, delta analysis, and optional trade plan. - SignalOutput: The structured output contract containing ticker, timestamp, price, heuristic verdict/confidence/S_total, probabilistic verdict/P_up/entropy/EV_R, delta analysis, and optional trade plan.
- Verdict: A pipeline decision of BUY, WATCH, or SKIP with associated confidence and reasoning.
- Confluence: The condition where a signal triggers across multiple timeframes; requires activation on at least 2 timeframes including at least one of D, W, or M.
- Entropy_Gate: Shannon entropy threshold used in the probabilistic pipeline to detect high-uncertainty states and force SKIP verdicts.
- EV_R: Expected value per unit of risk, computed as
P_up · E[win_R] - (1 - P_up) · 1.0, used as a quality gate in the probabilistic pipeline. - Signal_Cluster: A grouping of correlated signals (momentum, structure, volatility, fundamentals) used to prevent likelihood ratio stacking inflation in the Bayesian pipeline.
- Likelihood_Ratio: The ratio
P(signal|up) / P(signal|down)used in Bayesian updating, whereP(sig|up) = h·s + (1-h)·(1-s)·0.5. - Regime_Prior: The initial probability assigned based on market regime classification: bull=0.58, range=0.50, bear=0.42.
- OHLCV: Open, High, Low, Close, Volume — standard market data bar format.
- ATR: Average True Range — a volatility measure used for trailing stop calculations.
- Fibonacci_Retracement: A technical analysis tool computing price levels as
L(r) = SH - r·(SH - SL)where SH is swing high, SL is swing low, and r is a retracement ratio (0.236, 0.382, 0.5, 0.618, 0.786).
Requirements
Requirement 1: Input Normalization
User Story: As a signal engine operator, I want all market data, fundamentals, macro context, and open positions normalized into a single input structure, so that both pipelines consume identical inputs per evaluation tick.
Acceptance Criteria
- WHEN an evaluation tick is triggered for a ticker, THE Input_Normalizer SHALL construct a
NormalizedInputcontaining multi-timeframe OHLCV bars (M30, H1, H4, D, W, M), fundamental metrics (valuation_score, earnings_proximity_days), macro context (macro_bias as float in [-1.0, 1.0]), and open position state (entry_price, current_price, stop_loss, targets). - THE Input_Normalizer SHALL source OHLCV data from the existing market data tables, fundamental metrics from the existing company and trend data, and macro context from the existing
macro_impact_recordsandglobal_eventstables. - IF any required data source is unavailable or returns an error, THEN THE Input_Normalizer SHALL populate the corresponding field with a sentinel value (
Nonefor optional fields, empty list for OHLCV bars) and log a warning identifying the missing source. - THE Input_Normalizer SHALL validate that all OHLCV bars have monotonically increasing timestamps within each timeframe series.
- THE Input_Normalizer SHALL produce identical
NormalizedInputinstances for both pipelines within the same evaluation tick (shared reference, no independent fetches).
Requirement 2: Signal Library — Technical Signal Evaluation
User Story: As a quantitative analyst, I want a library of technical signal evaluators that produce scored signals per timeframe, so that both pipelines can consume standardized signal assessments.
Acceptance Criteria
- THE Signal_Library SHALL implement Fibonacci retracement signal evaluation using the formula
L(r) = SH - r·(SH - SL)for retracement ratios [0.236, 0.382, 0.5, 0.618, 0.786], where SH is the swing high and SL is the swing low within the evaluation window. - THE Signal_Library SHALL implement moving average stack evaluation that detects bullish alignment (MA_10 > MA_20 > MA_50 > MA_200) and bearish alignment (MA_10 < MA_20 < MA_50 < MA_200), producing a signal strength proportional to the degree of alignment.
- THE Signal_Library SHALL implement RSI evaluation using the standard 14-period RSI formula, producing overbought signals (RSI > 70) and oversold signals (RSI < 30) with strength scaled by distance from the threshold.
- THE Signal_Library SHALL implement Cup & Handle pattern detection that identifies the cup formation (U-shaped price recovery) and handle (small consolidation), producing a signal with confidence proportional to pattern completeness.
- THE Signal_Library SHALL implement Elliott Wave detection that identifies impulse waves (5-wave structure) and corrective waves (3-wave structure), producing a signal with the current wave position and projected direction.
- WHEN a signal evaluator receives insufficient data for its calculation (fewer bars than the required lookback period), THE Signal_Library SHALL return a null signal with a reason code indicating insufficient data rather than producing a partial evaluation.
- FOR ALL signal evaluators, THE Signal_Library SHALL produce output conforming to a common
SignalResultstructure containing: signal_type, timeframe, strength (float in [0.0, 1.0]), direction (bullish/bearish/neutral), confidence (float in [0.0, 1.0]), and metadata specific to the signal type.
Requirement 3: Multi-Timeframe Confluence Engine
User Story: As a quantitative analyst, I want signals evaluated across multiple timeframes with weighted confluence scoring, so that the engine prioritizes signals confirmed across longer timeframes.
Acceptance Criteria
- THE Multi_Timeframe_Engine SHALL evaluate each signal type across six timeframes with the following weights: M30=0.03, H1=0.07, H4=0.15, D=0.30, W=0.30, M=0.15.
- THE Multi_Timeframe_Engine SHALL compute a weighted confluence score as
C_confluence = Σ(w_tf · s_tf)wherew_tfis the timeframe weight ands_tfis the signal strength on that timeframe (0.0 if the signal did not trigger). - WHEN a signal triggers on fewer than 2 timeframes, THE Multi_Timeframe_Engine SHALL discard the signal from further pipeline processing (minimum confluence threshold).
- WHEN a signal triggers on 2 or more timeframes but none of D, W, or M are included, THE Multi_Timeframe_Engine SHALL discard the signal from further pipeline processing (higher-timeframe anchor requirement).
- THE Multi_Timeframe_Engine SHALL pass the confluence-filtered signals and their weighted scores to both the Heuristic_Pipeline and Probabilistic_Pipeline.
- FOR ALL signal sets where a signal triggers on more timeframes with higher weights, THE Multi_Timeframe_Engine SHALL produce a higher confluence score (monotonicity with respect to timeframe activation count and weight).
Requirement 4: Hard Filter Engine — Pre-Pipeline Gating
User Story: As a risk manager, I want hard filters that short-circuit both pipelines before evaluation, so that clearly unfavorable conditions produce immediate SKIP verdicts without wasting computation.
Acceptance Criteria
- WHEN the macro_bias value from the NormalizedInput equals -1.0, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "macro_bias_negative".
- WHEN the valuation_score from the NormalizedInput is below 0.3, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "valuation_below_threshold".
- WHEN the earnings_proximity_days from the NormalizedInput is 5 or fewer, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "earnings_block".
- WHEN multiple hard filters trigger simultaneously, THE Hard_Filter_Engine SHALL record all triggered filter reasons in the SKIP verdict (not just the first).
- WHEN no hard filters trigger, THE Hard_Filter_Engine SHALL pass the NormalizedInput through to both pipelines without modification.
- THE Hard_Filter_Engine SHALL execute before either pipeline begins evaluation, and both pipelines SHALL receive the same filter decision.
Requirement 5: Heuristic Pipeline — Deterministic Scoring and Verdict
User Story: As a quantitative analyst, I want the heuristic pipeline to produce a deterministic BUY/WATCH/SKIP verdict based on composite scoring of company, macro, and competitive signals, so that the system maintains a transparent, auditable scoring path.
Acceptance Criteria
- THE Heuristic_Pipeline SHALL compute a total score
S_total = S_company + S_macro + S_competitiveusing the existing three-layer signal aggregation with the currentWeightedSignalabstraction. - THE Heuristic_Pipeline SHALL compute signal weights using the formula
W_signal = gate · recency · credibility · (1 + novelty) · market_contextconsistent with the existingcompute_signal_weightfunction inscoring.py. - THE Heuristic_Pipeline SHALL compute a confidence value from the existing trend confidence formula incorporating source count, extraction confidence, signal agreement, and contradiction penalty.
- THE Heuristic_Pipeline SHALL produce a BUY verdict WHEN confidence >= 0.70 AND S_total >= 1.2 AND valuation_score >= 0.5 AND macro_bias > 0 AND earnings_proximity_days > 5.
- THE Heuristic_Pipeline SHALL produce a WATCH verdict WHEN confidence >= 0.55 AND the BUY conditions are not fully met.
- THE Heuristic_Pipeline SHALL produce a SKIP verdict WHEN confidence < 0.55.
- THE Heuristic_Pipeline SHALL emit a
HeuristicResultcontaining: verdict (BUY/WATCH/SKIP), confidence (float), S_total (float), S_company (float), S_macro (float), S_competitive (float), signal_weights (list), and reasoning (list of strings explaining the verdict).
Requirement 6: Probabilistic Pipeline — Bayesian Inference and Verdict
User Story: As a quantitative analyst, I want the probabilistic pipeline to produce a Bayesian BUY/WATCH/SKIP verdict using regime-based priors, likelihood ratios, entropy gating, and expected value calculation, so that the system captures uncertainty structure and risk-adjusted expected outcomes.
Acceptance Criteria
- THE Probabilistic_Pipeline SHALL initialize the prior probability based on the current market regime classification: bull regime → P_prior = 0.58, range regime → P_prior = 0.50, bear regime → P_prior = 0.42.
- THE Probabilistic_Pipeline SHALL compute likelihood ratios for each signal using
P(sig|up) = h·s + (1-h)·(1-s)·0.5andLR = P(sig|up) / P(sig|down), where h is the signal's historical hit rate and s is the signal strength. - THE Probabilistic_Pipeline SHALL update the posterior using log-odds accumulation:
logit(P_post) = logit(P_prior) + Σ log(LR_i), converting back to probability via the sigmoid function. - THE Probabilistic_Pipeline SHALL compute Shannon entropy
H = -P_up·log₂(P_up) - (1-P_up)·log₂(1-P_up)and apply entropy gating: WHEN H > 0.95, THE Probabilistic_Pipeline SHALL force a SKIP verdict with reason "high_entropy". - THE Probabilistic_Pipeline SHALL compute expected value per unit risk as
EV_R = P_up · E[win_R] - (1 - P_up) · 1.0whereE[win_R]is the expected win in risk units derived from signal strength and historical reward-risk ratios. - THE Probabilistic_Pipeline SHALL produce a BUY verdict WHEN P_up >= 0.60 AND entropy <= 0.90 AND EV_R >= 1.5 AND macro_bias > 0 AND valuation_score >= 0.5.
- THE Probabilistic_Pipeline SHALL produce a WATCH verdict WHEN P_up >= 0.55 AND entropy <= 0.95 AND the BUY conditions are not fully met.
- THE Probabilistic_Pipeline SHALL produce a SKIP verdict in all other cases.
- THE Probabilistic_Pipeline SHALL emit a
ProbabilisticResultcontaining: verdict (BUY/WATCH/SKIP), P_up (float), entropy (float), EV_R (float), prior (float), posterior (float), likelihood_ratios (list), regime (string), and reasoning (list of strings).
Requirement 7: Signal Correlation Penalty — Preventing LR Stacking Inflation
User Story: As a quantitative analyst, I want correlated signals grouped into clusters with a correlation penalty applied to prevent likelihood ratio stacking inflation, so that the Bayesian pipeline does not overstate confidence from redundant signals.
Acceptance Criteria
- THE Probabilistic_Pipeline SHALL classify each signal into one of four clusters: momentum (MA stack, RSI), structure (Fibonacci retracement, Elliott Wave), volatility (ATR-based signals, Bollinger-derived), and fundamentals (valuation, earnings, macro).
- WHEN multiple signals within the same cluster produce likelihood ratios in the same direction, THE Probabilistic_Pipeline SHALL apply a within-cluster penalty: only the strongest LR in the cluster contributes at full weight, and subsequent LRs in the same cluster contribute at a decay factor of 0.5^(n-1) where n is the signal's rank within the cluster by LR magnitude.
- THE Probabilistic_Pipeline SHALL apply no penalty across different clusters (signals from different clusters are treated as independent).
- WHEN a cluster contains only one signal, THE Probabilistic_Pipeline SHALL apply no penalty to that signal.
- FOR ALL signal sets, THE Probabilistic_Pipeline SHALL produce a posterior probability that is less than or equal to the posterior computed without the correlation penalty (the penalty only reduces confidence, never inflates it).
Requirement 8: Exit Engine — Position Management
User Story: As a trader, I want the signal engine to evaluate exit conditions for open positions, so that stop hits, take-profit targets, and trailing stops are managed as part of the signal evaluation cycle.
Acceptance Criteria
- WHEN the current price of an open position hits or crosses below the stop_loss level, THE Exit_Engine SHALL emit an EXIT_FULL signal for that position with reason "stop_hit".
- WHEN the current price of an open position hits or crosses above the first take-profit target (target_1), THE Exit_Engine SHALL emit an EXIT_HALF signal for that position with reason "target_1_hit".
- WHEN the current price of an open position hits or crosses above the second take-profit target (target_2), THE Exit_Engine SHALL emit an EXIT_FULL signal for that position with reason "target_2_hit".
- WHEN a partial exit has been executed (EXIT_HALF), THE Exit_Engine SHALL activate a trailing stop at
current_price - ATR · trailing_multiplierand update the trailing stop upward as the price advances (the trailing stop moves up but does not move down). - WHEN the trailing stop is active and the current price crosses below the trailing stop level, THE Exit_Engine SHALL emit an EXIT_FULL signal for the remaining position with reason "trailing_stop_hit".
- THE Exit_Engine SHALL evaluate exit conditions before the signal pipelines run for new entry signals, so that exit signals take priority over new entry signals for the same ticker.
- THE Exit_Engine SHALL emit exit signals as part of the
SignalOutputcontract with the position identifier, exit type (EXIT_HALF/EXIT_FULL), and reason.
Requirement 9: Delta Analyzer — Pipeline Agreement Tracking
User Story: As a model developer, I want the delta analyzer to compare heuristic and probabilistic verdicts and record disagreement details, so that I can generate training signals for future model tuning.
Acceptance Criteria
- WHEN both pipelines produce verdicts for the same ticker and tick, THE Delta_Analyzer SHALL compute an agreement flag (true if both verdicts are identical, false otherwise).
- THE Delta_Analyzer SHALL compute a confidence delta as
|heuristic_confidence - probabilistic_P_up|representing the magnitude of disagreement between the two pipelines. - WHEN the pipelines disagree on verdict, THE Delta_Analyzer SHALL record the disagreement reason by identifying which conditions differed (e.g., "heuristic_confidence_below_threshold", "probabilistic_entropy_too_high", "EV_R_below_threshold").
- THE Delta_Analyzer SHALL track a rolling agreement rate over the last 100 evaluations per ticker, stored in Redis for dashboard consumption.
- THE Delta_Analyzer SHALL emit a
DeltaResultcontaining: agreement (bool), confidence_delta (float), heuristic_verdict (string), probabilistic_verdict (string), disagreement_reasons (list of strings), and rolling_agreement_rate (float). - WHEN the rolling agreement rate drops below 0.50 for a ticker, THE Delta_Analyzer SHALL log a warning indicating persistent pipeline disagreement for operator review.
Requirement 10: Output Formatter — Structured SignalOutput Contract
User Story: As a downstream system consumer, I want the signal engine to emit a structured SignalOutput contract, so that the trading engine, delta analysis dashboard, and audit systems can consume a consistent output format.
Acceptance Criteria
- THE Output_Formatter SHALL produce a
SignalOutputcontaining: ticker (string), timestamp (datetime), price (float), heuristic section (verdict, confidence, S_total), probabilistic section (verdict, P_up, entropy, EV_R), delta section (agreement, confidence_delta, disagreement_reasons), and optional trade_plan section. - WHEN the heuristic pipeline produces a BUY verdict, THE Output_Formatter SHALL populate the trade_plan section with entry_price, stop_loss, target_1, target_2, and position_size derived from the heuristic confidence and existing position sizing logic.
- WHEN the probabilistic pipeline produces a BUY verdict but the heuristic pipeline does not, THE Output_Formatter SHALL populate the trade_plan section with a "probabilistic_only" flag and reduced position sizing (50% of standard).
- WHEN both pipelines produce a BUY verdict, THE Output_Formatter SHALL populate the trade_plan section with full position sizing and a "dual_confirmed" flag.
- THE Output_Formatter SHALL serialize the
SignalOutputas a Pydantic model with JSON serialization support for Redis queue publishing and database persistence. - FOR ALL valid pipeline results, THE Output_Formatter SHALL produce a
SignalOutputthat round-trips through JSON serialization and deserialization without data loss (parse(format(output)) produces an equivalent object).
Requirement 11: Dual Pipeline Orchestration
User Story: As a signal engine operator, I want both pipelines to run concurrently per evaluation tick sharing the same inputs, so that the system produces independent verdicts without redundant data fetching.
Acceptance Criteria
- WHEN an evaluation tick is triggered, THE Signal_Engine SHALL execute the Input_Normalizer once, then pass the resulting
NormalizedInputto the Hard_Filter_Engine, then (if not filtered) execute both the Heuristic_Pipeline and Probabilistic_Pipeline concurrently usingasyncio.gather. - THE Signal_Engine SHALL enforce that both pipelines receive identical
NormalizedInputreferences (no independent data fetches that could produce different snapshots). - WHEN either pipeline raises an exception during evaluation, THE Signal_Engine SHALL catch the exception, log the error with full traceback, and produce a SKIP verdict for the failed pipeline with reason "pipeline_error" while allowing the other pipeline to complete normally.
- THE Signal_Engine SHALL measure and log the wall-clock execution time of each pipeline per tick for performance monitoring.
- THE Signal_Engine SHALL publish the assembled
SignalOutputto the existing Redis queue (stonks:queue:trading_decisions) for consumption by the trading engine. - THE Signal_Engine SHALL persist each
SignalOutputto a database table for historical analysis and audit.
Requirement 12: Integration with Existing Trading Engine
User Story: As a platform operator, I want the dual-pipeline signal engine to integrate with the existing trading engine, so that the trading engine can consume SignalOutput verdicts and make execution decisions.
Acceptance Criteria
- THE Signal_Engine SHALL publish
SignalOutputto the existingstonks:queue:trading_decisionsRedis queue in a format compatible with the existingTradingEngine.evaluate_recommendationinterface. - THE Signal_Engine SHALL map the
SignalOutputtrade_plan to the existingRecommendationschema fields (action, confidence, position_sizing) so that the trading engine can process dual-pipeline outputs without modification to its core evaluation logic. - WHEN the
SignalOutputhas a "dual_confirmed" flag, THE Signal_Engine SHALL set the recommendation confidence to the maximum of heuristic_confidence and probabilistic_P_up. - WHEN the
SignalOutputhas a "probabilistic_only" flag, THE Signal_Engine SHALL set the recommendation confidence toprobabilistic_P_up · 0.8(20% confidence haircut for single-pipeline confirmation). - WHEN neither pipeline produces a BUY verdict, THE Signal_Engine SHALL not publish a trading recommendation to the queue (WATCH and SKIP verdicts are persisted for analysis but not forwarded to the trading engine).
Requirement 13: Configuration and Feature Flags
User Story: As a platform operator, I want the dual-pipeline engine configurable via the existing risk_configs table and environment variables, so that I can tune thresholds, enable/disable individual pipelines, and adjust timeframe weights without code changes.
Acceptance Criteria
- THE Signal_Engine SHALL support a
dual_pipeline_enabledfeature flag inrisk_configsthat toggles the entire dual-pipeline engine on or off, defaulting to false for safe rollout. - THE Signal_Engine SHALL support independent enable/disable flags for each pipeline:
heuristic_pipeline_enabledandprobabilistic_pipeline_enabled, both defaulting to true when the dual-pipeline engine is enabled. - THE Signal_Engine SHALL support configurable timeframe weights via a
timeframe_weightsJSON object inrisk_configs, defaulting to{"M30": 0.03, "H1": 0.07, "H4": 0.15, "D": 0.30, "W": 0.30, "M": 0.15}. - THE Signal_Engine SHALL support configurable hard filter thresholds:
hard_filter_valuation_min(default 0.3),hard_filter_earnings_days(default 5), andhard_filter_macro_bias_skip(default -1.0). - THE Signal_Engine SHALL support configurable verdict thresholds for both pipelines via
risk_configsJSON, including heuristic confidence thresholds (BUY: 0.70, WATCH: 0.55) and probabilistic thresholds (P_up: 0.60, entropy: 0.90, EV_R: 1.5). - IF the
dual_pipeline_enabledflag fails to read from the database, THEN THE Signal_Engine SHALL default to disabled (fail-safe behavior) and log a warning. - THE Signal_Engine SHALL log the active configuration at startup and on each configuration change for auditability.
Requirement 14: Regime-Based Prior Engine
User Story: As a quantitative analyst, I want the probabilistic pipeline's prior probability to adapt based on the current market regime, so that the Bayesian inference starts from a regime-appropriate baseline rather than a fixed 0.50.
Acceptance Criteria
- THE Probabilistic_Pipeline SHALL use the existing
classify_regimefunction fromservices/aggregation/regime.pyto determine the current market regime for each ticker. - THE Probabilistic_Pipeline SHALL map regime classifications to prior probabilities: trend_following with positive trend_indicator → 0.58 (bull), trend_following with negative trend_indicator → 0.42 (bear), mean_reversion → 0.50 (range), panic → 0.42 (bear), uncertainty → 0.50 (range).
- THE Probabilistic_Pipeline SHALL convert the regime prior to log-odds before accumulating likelihood ratios:
logit(P_prior) = log(P_prior / (1 - P_prior)). - WHEN market data is insufficient for regime classification (fewer than 100 days of price history), THE Probabilistic_Pipeline SHALL use the uncertainty prior of 0.50.
- THE Probabilistic_Pipeline SHALL record the regime classification and prior probability in the
ProbabilisticResultfor auditability.
Requirement 15: Database Schema for Signal Engine Output
User Story: As a platform operator, I want signal engine outputs persisted to a dedicated database table, so that historical evaluations are available for analysis, backtesting, and audit.
Acceptance Criteria
- THE Signal_Engine SHALL persist each
SignalOutputto asignal_engine_outputstable with columns for: id (UUID primary key), ticker (text), evaluated_at (timestamptz), price (numeric), heuristic_verdict (text), heuristic_confidence (numeric), heuristic_s_total (numeric), probabilistic_verdict (text), probabilistic_p_up (numeric), probabilistic_entropy (numeric), probabilistic_ev_r (numeric), delta_agreement (boolean), delta_confidence_delta (numeric), delta_reasons (JSONB), trade_plan (JSONB), full_output (JSONB), created_at (timestamptz). - THE Signal_Engine SHALL create an index on
(ticker, evaluated_at)for efficient time-range queries per ticker. - THE Signal_Engine SHALL create an index on
evaluated_atfor efficient global time-range queries. - WHEN persisting fails due to a database error, THE Signal_Engine SHALL log the error and continue processing (persistence failure does not block signal emission to the trading queue).
Requirement 16: Backward Compatibility and Migration Path
User Story: As a platform operator, I want the dual-pipeline engine to coexist with the existing single-pipeline aggregation, so that the rollout is incremental and reversible.
Acceptance Criteria
- WHEN
dual_pipeline_enabledis false, THE Signal_Engine SHALL not run, and the existing aggregation pipeline SHALL continue to operate unchanged. - WHEN
dual_pipeline_enabledis true, THE Signal_Engine SHALL run alongside the existing aggregation pipeline, with the trading engine consumingSignalOutputfrom the dual-pipeline engine instead ofRecommendationfrom the existing recommendation worker. - THE Signal_Engine SHALL reuse the existing
WeightedSignal,BayesianPosterior,RegimeClassification, andTrendSummarydata structures fromservices/aggregation/rather than duplicating them. - THE Signal_Engine SHALL reuse the existing
compute_signal_weight,compute_bayesian_posterior, andclassify_regimefunctions rather than reimplementing the underlying math. - THE Signal_Engine SHALL add the new
signal_engine_outputstable via a new database migration without modifying existing tables. - THE Signal_Engine SHALL support running in "shadow mode" where both the existing pipeline and the dual-pipeline engine run, but only the existing pipeline's output is forwarded to the trading engine (dual-pipeline output is persisted for comparison only).
Requirement 17: Property-Based Testing for Dual-Pipeline Correctness
User Story: As a developer, I want comprehensive property-based tests validating the mathematical correctness and structural invariants of the dual-pipeline engine, so that edge cases and numerical stability issues are caught before deployment.
Acceptance Criteria
- THE test suite SHALL include property-based tests for the Fibonacci retracement formula verifying that
L(r) = SH - r·(SH - SL)produces values in [SL, SH] for all r in [0, 1] and all SH > SL > 0. - THE test suite SHALL include property-based tests for the Bayesian log-odds update verifying that
logit(P_post) = logit(P_prior) + Σ log(LR_i)round-trips correctly: converting P_prior to logit, adding log-LRs, and converting back via sigmoid produces a valid probability in (0, 1). - THE test suite SHALL include property-based tests for the entropy gate verifying that Shannon entropy is maximized at P_up = 0.5 and equals 0.0 at P_up = 0.0 or P_up = 1.0, and is symmetric around 0.5.
- THE test suite SHALL include property-based tests for the signal correlation penalty verifying that the penalized posterior is always less than or equal to the unpenalized posterior for any signal set with correlated signals.
- THE test suite SHALL include property-based tests for the multi-timeframe confluence score verifying monotonicity: activating a signal on an additional timeframe with non-zero weight always increases or maintains the confluence score.
- THE test suite SHALL include property-based tests for the
SignalOutputcontract verifying round-trip serialization:SignalOutput.model_validate_json(output.model_dump_json())produces an equivalent object for all valid outputs. - THE test suite SHALL include property-based tests for the hard filter engine verifying that macro_bias = -1.0 always produces SKIP, valuation_score < 0.3 always produces SKIP, and earnings_proximity_days <= 5 always produces SKIP, regardless of all other input values.
- THE test suite SHALL include property-based tests for the EV_R calculation verifying that
EV_R = P_up · E[win_R] - (1 - P_up) · 1.0is monotonically increasing with P_up for fixed E[win_R] > 0.