feat: implement dual-pipeline signal engine service
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled

New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.

Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
  gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading

Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant

Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)
This commit is contained in:
Celes Renata
2026-05-02 07:32:26 +00:00
parent 7e2343ec2c
commit f468e30af0
61 changed files with 14107 additions and 184 deletions
+299
View File
@@ -0,0 +1,299 @@
"""Heuristic Pipeline (Pipeline A) — Deterministic scoring and verdict.
Computes ``S_total = S_company + S_macro + S_competitive`` from confluence-
filtered signals and produces a confidence-gated BUY / WATCH / SKIP verdict.
The pipeline reuses the existing ``compute_signal_weight`` infrastructure
from ``services.aggregation.scoring`` for signal weighting and follows the
three-layer signal aggregation model (company, macro, competitive).
Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7
"""
from __future__ import annotations
import logging
from services.signal_engine.config import HeuristicConfig
from services.signal_engine.models import (
ConfluenceSignal,
HeuristicResult,
NormalizedInput,
SignalDirection,
Verdict,
)
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Signal classification — which confluence signals belong to which layer
# ---------------------------------------------------------------------------
# Company-level technical signals (Layer 1)
COMPANY_SIGNAL_TYPES: frozenset[str] = frozenset({
"fibonacci",
"ma_stack",
"rsi",
"cup_handle",
"elliott_wave",
})
# Competitive signals (Layer 3) — future expansion
COMPETITIVE_SIGNAL_TYPES: frozenset[str] = frozenset()
# Macro weight applied to macro_bias to produce S_macro
_MACRO_WEIGHT: float = 0.5
# ---------------------------------------------------------------------------
# Score computation helpers
# ---------------------------------------------------------------------------
def _compute_s_company(confluence_signals: list[ConfluenceSignal]) -> tuple[float, list[dict]]:
"""Sum confluence scores for company-level signals.
Returns the total S_company score and a list of per-signal weight
breakdowns for audit.
"""
s_company = 0.0
weights: list[dict] = []
for sig in confluence_signals:
if sig.signal_type in COMPANY_SIGNAL_TYPES:
# Direction-aware: bullish contributes positively, bearish negatively
direction_sign = _direction_sign(sig.direction)
contribution = sig.confluence_score * direction_sign
s_company += contribution
weights.append({
"signal_type": sig.signal_type,
"layer": "company",
"confluence_score": sig.confluence_score,
"direction": sig.direction.value,
"contribution": contribution,
"active_timeframes": sig.active_timeframes,
})
return s_company, weights
def _compute_s_macro(normalized: NormalizedInput) -> float:
"""Compute macro score from macro_bias.
S_macro = macro_bias * weight, where macro_bias is in [-1.0, 1.0].
A positive macro_bias contributes positively; negative contributes
negatively.
"""
return normalized.macro_bias * _MACRO_WEIGHT
def _compute_s_competitive(confluence_signals: list[ConfluenceSignal]) -> float:
"""Sum confluence scores for competitive-layer signals.
Currently returns 0.0 as no competitive signal types are defined in
the signal library. This is a placeholder for future expansion.
"""
s_competitive = 0.0
for sig in confluence_signals:
if sig.signal_type in COMPETITIVE_SIGNAL_TYPES:
direction_sign = _direction_sign(sig.direction)
s_competitive += sig.confluence_score * direction_sign
return s_competitive
def _direction_sign(direction: SignalDirection) -> float:
"""Map signal direction to a numeric sign."""
if direction == SignalDirection.BULLISH:
return 1.0
if direction == SignalDirection.BEARISH:
return -1.0
return 0.0
# ---------------------------------------------------------------------------
# Confidence computation
# ---------------------------------------------------------------------------
def _compute_confidence(
confluence_signals: list[ConfluenceSignal],
) -> float:
"""Compute pipeline confidence from confluence signals.
Confidence is derived from:
1. **Base confidence** — average signal strength across all confluence
signals (mean of confluence_score values).
2. **Source count boost** — more active signals increase confidence
(diminishing returns, capped contribution).
3. **Signal agreement boost** — if all signals point in the same
direction, confidence is boosted.
4. **Contradiction penalty** — if signals disagree on direction,
confidence is penalised.
Returns a value clamped to [0.0, 1.0].
"""
if not confluence_signals:
return 0.0
# 1. Base confidence: average confluence score (already weighted by
# timeframe importance)
total_score = sum(s.confluence_score for s in confluence_signals)
base_confidence = total_score / len(confluence_signals)
# 2. Source count factor: more signals → higher confidence, with
# diminishing returns. 1 signal → 0.6, 2 → 0.75, 3 → 0.85,
# 4 → 0.90, 5+ → 0.95 (asymptotic).
n = len(confluence_signals)
source_factor = 1.0 - (0.4 / n) # approaches 1.0 as n grows
# 3. Signal agreement / contradiction
directions = [s.direction for s in confluence_signals]
bullish_count = sum(1 for d in directions if d == SignalDirection.BULLISH)
bearish_count = sum(1 for d in directions if d == SignalDirection.BEARISH)
if n == 1:
agreement_factor = 1.0
elif bullish_count == n or bearish_count == n:
# Perfect agreement — boost
agreement_factor = 1.15
elif bullish_count > 0 and bearish_count > 0:
# Contradiction — penalty proportional to minority fraction
minority = min(bullish_count, bearish_count)
contradiction_ratio = minority / n
agreement_factor = 1.0 - (0.3 * contradiction_ratio)
else:
# Mix of directional and neutral — mild boost
agreement_factor = 1.05
confidence = base_confidence * source_factor * agreement_factor
return max(0.0, min(confidence, 1.0))
# ---------------------------------------------------------------------------
# Verdict logic
# ---------------------------------------------------------------------------
def _determine_verdict(
confidence: float,
s_total: float,
normalized: NormalizedInput,
config: HeuristicConfig,
) -> tuple[Verdict, list[str]]:
"""Apply threshold logic to determine BUY / WATCH / SKIP verdict.
Returns the verdict and a list of reasoning strings explaining the
decision.
"""
reasoning: list[str] = []
valuation_score = normalized.valuation_score if normalized.valuation_score is not None else 0.0
earnings_days = normalized.earnings_proximity_days if normalized.earnings_proximity_days is not None else 0
# --- Check BUY conditions ---
buy_conditions = {
"confidence": confidence >= config.buy_confidence,
"s_total": s_total >= config.buy_s_total,
"valuation": valuation_score >= config.buy_valuation_min,
"macro_bias": normalized.macro_bias > config.macro_bias_threshold,
"earnings_proximity": earnings_days > config.earnings_days_threshold,
}
all_buy_met = all(buy_conditions.values())
if all_buy_met:
reasoning.append(
f"BUY: all conditions met — confidence={confidence:.3f} "
f"(>= {config.buy_confidence}), S_total={s_total:.3f} "
f"(>= {config.buy_s_total}), valuation={valuation_score:.2f} "
f"(>= {config.buy_valuation_min}), macro_bias={normalized.macro_bias:.2f} "
f"(> {config.macro_bias_threshold}), earnings_days={earnings_days} "
f"(> {config.earnings_days_threshold})"
)
return Verdict.BUY, reasoning
# --- Check WATCH conditions ---
if confidence >= config.watch_confidence:
# WATCH: confidence is sufficient but not all BUY conditions met
failed_conditions = [k for k, v in buy_conditions.items() if not v]
reasoning.append(
f"WATCH: confidence={confidence:.3f} (>= {config.watch_confidence}) "
f"but BUY conditions not fully met — failed: {', '.join(failed_conditions)}"
)
for cond_name, met in buy_conditions.items():
if not met:
reasoning.append(f" - {cond_name} not met")
return Verdict.WATCH, reasoning
# --- SKIP ---
reasoning.append(
f"SKIP: confidence={confidence:.3f} < {config.watch_confidence} "
f"(watch threshold)"
)
return Verdict.SKIP, reasoning
# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------
def run_heuristic_pipeline(
normalized: NormalizedInput,
confluence_signals: list[ConfluenceSignal],
config: HeuristicConfig,
) -> HeuristicResult:
"""Run the deterministic heuristic pipeline.
Computes ``S_total = S_company + S_macro + S_competitive`` using the
existing three-layer signal aggregation model and produces a
confidence-gated BUY / WATCH / SKIP verdict.
Args:
normalized: The unified input structure for this evaluation tick.
confluence_signals: Signals that passed multi-timeframe confluence
filtering.
config: Heuristic pipeline thresholds.
Returns:
A :class:`HeuristicResult` with verdict, scores, weights, and
reasoning.
Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7
"""
# 1. Compute three-layer scores
s_company, signal_weights = _compute_s_company(confluence_signals)
s_macro = _compute_s_macro(normalized)
s_competitive = _compute_s_competitive(confluence_signals)
s_total = s_company + s_macro + s_competitive
# 2. Compute confidence
confidence = _compute_confidence(confluence_signals)
# 3. Determine verdict
verdict, reasoning = _determine_verdict(confidence, s_total, normalized, config)
logger.info(
"Heuristic pipeline [%s]: verdict=%s confidence=%.3f "
"S_total=%.3f (company=%.3f macro=%.3f competitive=%.3f) "
"signals=%d",
normalized.ticker,
verdict.value,
confidence,
s_total,
s_company,
s_macro,
s_competitive,
len(confluence_signals),
)
return HeuristicResult(
verdict=verdict,
confidence=confidence,
s_total=s_total,
s_company=s_company,
s_macro=s_macro,
s_competitive=s_competitive,
signal_weights=signal_weights,
reasoning=reasoning,
)