feat: implement dual-pipeline signal engine service
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
New service at services/signal_engine/ implementing concurrent heuristic (deterministic scoring) and probabilistic (Bayesian inference) pipelines that evaluate technical signals across 6 timeframes (M30-M) and produce independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick. Components: - Input Normalizer: multi-source data assembly with sentinel fallbacks - Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave - Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors - Hard Filter Engine: macro_bias, valuation, earnings proximity gating - Heuristic Pipeline: S_total scoring with confidence-gated verdicts - Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy gating, EV_R calculation, and signal correlation penalty - Exit Engine: stop-loss, targets, trailing ATR-based stops - Delta Analyzer: pipeline agreement tracking with rolling Redis metrics - Output Formatter: SignalOutput contract + Recommendation schema mapping - Worker orchestrator: concurrent pipelines with failure isolation - Main entry point: queue polling with fail-safe config loading Infrastructure: - Migration 039: signal_engine_outputs table with 3 indexes - Helm chart: signalEngine service entry (processing tier) - Redis key: QUEUE_SIGNAL_ENGINE constant Tests: 390 tests (unit + property-based) covering all components Config: dual_pipeline_enabled=false by default (safe rollout)
This commit is contained in:
@@ -0,0 +1,299 @@
|
||||
"""Heuristic Pipeline (Pipeline A) — Deterministic scoring and verdict.
|
||||
|
||||
Computes ``S_total = S_company + S_macro + S_competitive`` from confluence-
|
||||
filtered signals and produces a confidence-gated BUY / WATCH / SKIP verdict.
|
||||
|
||||
The pipeline reuses the existing ``compute_signal_weight`` infrastructure
|
||||
from ``services.aggregation.scoring`` for signal weighting and follows the
|
||||
three-layer signal aggregation model (company, macro, competitive).
|
||||
|
||||
Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from services.signal_engine.config import HeuristicConfig
|
||||
from services.signal_engine.models import (
|
||||
ConfluenceSignal,
|
||||
HeuristicResult,
|
||||
NormalizedInput,
|
||||
SignalDirection,
|
||||
Verdict,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Signal classification — which confluence signals belong to which layer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Company-level technical signals (Layer 1)
|
||||
COMPANY_SIGNAL_TYPES: frozenset[str] = frozenset({
|
||||
"fibonacci",
|
||||
"ma_stack",
|
||||
"rsi",
|
||||
"cup_handle",
|
||||
"elliott_wave",
|
||||
})
|
||||
|
||||
# Competitive signals (Layer 3) — future expansion
|
||||
COMPETITIVE_SIGNAL_TYPES: frozenset[str] = frozenset()
|
||||
|
||||
# Macro weight applied to macro_bias to produce S_macro
|
||||
_MACRO_WEIGHT: float = 0.5
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Score computation helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _compute_s_company(confluence_signals: list[ConfluenceSignal]) -> tuple[float, list[dict]]:
|
||||
"""Sum confluence scores for company-level signals.
|
||||
|
||||
Returns the total S_company score and a list of per-signal weight
|
||||
breakdowns for audit.
|
||||
"""
|
||||
s_company = 0.0
|
||||
weights: list[dict] = []
|
||||
|
||||
for sig in confluence_signals:
|
||||
if sig.signal_type in COMPANY_SIGNAL_TYPES:
|
||||
# Direction-aware: bullish contributes positively, bearish negatively
|
||||
direction_sign = _direction_sign(sig.direction)
|
||||
contribution = sig.confluence_score * direction_sign
|
||||
s_company += contribution
|
||||
weights.append({
|
||||
"signal_type": sig.signal_type,
|
||||
"layer": "company",
|
||||
"confluence_score": sig.confluence_score,
|
||||
"direction": sig.direction.value,
|
||||
"contribution": contribution,
|
||||
"active_timeframes": sig.active_timeframes,
|
||||
})
|
||||
|
||||
return s_company, weights
|
||||
|
||||
|
||||
def _compute_s_macro(normalized: NormalizedInput) -> float:
|
||||
"""Compute macro score from macro_bias.
|
||||
|
||||
S_macro = macro_bias * weight, where macro_bias is in [-1.0, 1.0].
|
||||
A positive macro_bias contributes positively; negative contributes
|
||||
negatively.
|
||||
"""
|
||||
return normalized.macro_bias * _MACRO_WEIGHT
|
||||
|
||||
|
||||
def _compute_s_competitive(confluence_signals: list[ConfluenceSignal]) -> float:
|
||||
"""Sum confluence scores for competitive-layer signals.
|
||||
|
||||
Currently returns 0.0 as no competitive signal types are defined in
|
||||
the signal library. This is a placeholder for future expansion.
|
||||
"""
|
||||
s_competitive = 0.0
|
||||
for sig in confluence_signals:
|
||||
if sig.signal_type in COMPETITIVE_SIGNAL_TYPES:
|
||||
direction_sign = _direction_sign(sig.direction)
|
||||
s_competitive += sig.confluence_score * direction_sign
|
||||
return s_competitive
|
||||
|
||||
|
||||
def _direction_sign(direction: SignalDirection) -> float:
|
||||
"""Map signal direction to a numeric sign."""
|
||||
if direction == SignalDirection.BULLISH:
|
||||
return 1.0
|
||||
if direction == SignalDirection.BEARISH:
|
||||
return -1.0
|
||||
return 0.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Confidence computation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _compute_confidence(
|
||||
confluence_signals: list[ConfluenceSignal],
|
||||
) -> float:
|
||||
"""Compute pipeline confidence from confluence signals.
|
||||
|
||||
Confidence is derived from:
|
||||
1. **Base confidence** — average signal strength across all confluence
|
||||
signals (mean of confluence_score values).
|
||||
2. **Source count boost** — more active signals increase confidence
|
||||
(diminishing returns, capped contribution).
|
||||
3. **Signal agreement boost** — if all signals point in the same
|
||||
direction, confidence is boosted.
|
||||
4. **Contradiction penalty** — if signals disagree on direction,
|
||||
confidence is penalised.
|
||||
|
||||
Returns a value clamped to [0.0, 1.0].
|
||||
"""
|
||||
if not confluence_signals:
|
||||
return 0.0
|
||||
|
||||
# 1. Base confidence: average confluence score (already weighted by
|
||||
# timeframe importance)
|
||||
total_score = sum(s.confluence_score for s in confluence_signals)
|
||||
base_confidence = total_score / len(confluence_signals)
|
||||
|
||||
# 2. Source count factor: more signals → higher confidence, with
|
||||
# diminishing returns. 1 signal → 0.6, 2 → 0.75, 3 → 0.85,
|
||||
# 4 → 0.90, 5+ → 0.95 (asymptotic).
|
||||
n = len(confluence_signals)
|
||||
source_factor = 1.0 - (0.4 / n) # approaches 1.0 as n grows
|
||||
|
||||
# 3. Signal agreement / contradiction
|
||||
directions = [s.direction for s in confluence_signals]
|
||||
bullish_count = sum(1 for d in directions if d == SignalDirection.BULLISH)
|
||||
bearish_count = sum(1 for d in directions if d == SignalDirection.BEARISH)
|
||||
|
||||
if n == 1:
|
||||
agreement_factor = 1.0
|
||||
elif bullish_count == n or bearish_count == n:
|
||||
# Perfect agreement — boost
|
||||
agreement_factor = 1.15
|
||||
elif bullish_count > 0 and bearish_count > 0:
|
||||
# Contradiction — penalty proportional to minority fraction
|
||||
minority = min(bullish_count, bearish_count)
|
||||
contradiction_ratio = minority / n
|
||||
agreement_factor = 1.0 - (0.3 * contradiction_ratio)
|
||||
else:
|
||||
# Mix of directional and neutral — mild boost
|
||||
agreement_factor = 1.05
|
||||
|
||||
confidence = base_confidence * source_factor * agreement_factor
|
||||
return max(0.0, min(confidence, 1.0))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Verdict logic
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _determine_verdict(
|
||||
confidence: float,
|
||||
s_total: float,
|
||||
normalized: NormalizedInput,
|
||||
config: HeuristicConfig,
|
||||
) -> tuple[Verdict, list[str]]:
|
||||
"""Apply threshold logic to determine BUY / WATCH / SKIP verdict.
|
||||
|
||||
Returns the verdict and a list of reasoning strings explaining the
|
||||
decision.
|
||||
"""
|
||||
reasoning: list[str] = []
|
||||
|
||||
valuation_score = normalized.valuation_score if normalized.valuation_score is not None else 0.0
|
||||
earnings_days = normalized.earnings_proximity_days if normalized.earnings_proximity_days is not None else 0
|
||||
|
||||
# --- Check BUY conditions ---
|
||||
buy_conditions = {
|
||||
"confidence": confidence >= config.buy_confidence,
|
||||
"s_total": s_total >= config.buy_s_total,
|
||||
"valuation": valuation_score >= config.buy_valuation_min,
|
||||
"macro_bias": normalized.macro_bias > config.macro_bias_threshold,
|
||||
"earnings_proximity": earnings_days > config.earnings_days_threshold,
|
||||
}
|
||||
|
||||
all_buy_met = all(buy_conditions.values())
|
||||
|
||||
if all_buy_met:
|
||||
reasoning.append(
|
||||
f"BUY: all conditions met — confidence={confidence:.3f} "
|
||||
f"(>= {config.buy_confidence}), S_total={s_total:.3f} "
|
||||
f"(>= {config.buy_s_total}), valuation={valuation_score:.2f} "
|
||||
f"(>= {config.buy_valuation_min}), macro_bias={normalized.macro_bias:.2f} "
|
||||
f"(> {config.macro_bias_threshold}), earnings_days={earnings_days} "
|
||||
f"(> {config.earnings_days_threshold})"
|
||||
)
|
||||
return Verdict.BUY, reasoning
|
||||
|
||||
# --- Check WATCH conditions ---
|
||||
if confidence >= config.watch_confidence:
|
||||
# WATCH: confidence is sufficient but not all BUY conditions met
|
||||
failed_conditions = [k for k, v in buy_conditions.items() if not v]
|
||||
reasoning.append(
|
||||
f"WATCH: confidence={confidence:.3f} (>= {config.watch_confidence}) "
|
||||
f"but BUY conditions not fully met — failed: {', '.join(failed_conditions)}"
|
||||
)
|
||||
for cond_name, met in buy_conditions.items():
|
||||
if not met:
|
||||
reasoning.append(f" - {cond_name} not met")
|
||||
return Verdict.WATCH, reasoning
|
||||
|
||||
# --- SKIP ---
|
||||
reasoning.append(
|
||||
f"SKIP: confidence={confidence:.3f} < {config.watch_confidence} "
|
||||
f"(watch threshold)"
|
||||
)
|
||||
return Verdict.SKIP, reasoning
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_heuristic_pipeline(
|
||||
normalized: NormalizedInput,
|
||||
confluence_signals: list[ConfluenceSignal],
|
||||
config: HeuristicConfig,
|
||||
) -> HeuristicResult:
|
||||
"""Run the deterministic heuristic pipeline.
|
||||
|
||||
Computes ``S_total = S_company + S_macro + S_competitive`` using the
|
||||
existing three-layer signal aggregation model and produces a
|
||||
confidence-gated BUY / WATCH / SKIP verdict.
|
||||
|
||||
Args:
|
||||
normalized: The unified input structure for this evaluation tick.
|
||||
confluence_signals: Signals that passed multi-timeframe confluence
|
||||
filtering.
|
||||
config: Heuristic pipeline thresholds.
|
||||
|
||||
Returns:
|
||||
A :class:`HeuristicResult` with verdict, scores, weights, and
|
||||
reasoning.
|
||||
|
||||
Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7
|
||||
"""
|
||||
# 1. Compute three-layer scores
|
||||
s_company, signal_weights = _compute_s_company(confluence_signals)
|
||||
s_macro = _compute_s_macro(normalized)
|
||||
s_competitive = _compute_s_competitive(confluence_signals)
|
||||
s_total = s_company + s_macro + s_competitive
|
||||
|
||||
# 2. Compute confidence
|
||||
confidence = _compute_confidence(confluence_signals)
|
||||
|
||||
# 3. Determine verdict
|
||||
verdict, reasoning = _determine_verdict(confidence, s_total, normalized, config)
|
||||
|
||||
logger.info(
|
||||
"Heuristic pipeline [%s]: verdict=%s confidence=%.3f "
|
||||
"S_total=%.3f (company=%.3f macro=%.3f competitive=%.3f) "
|
||||
"signals=%d",
|
||||
normalized.ticker,
|
||||
verdict.value,
|
||||
confidence,
|
||||
s_total,
|
||||
s_company,
|
||||
s_macro,
|
||||
s_competitive,
|
||||
len(confluence_signals),
|
||||
)
|
||||
|
||||
return HeuristicResult(
|
||||
verdict=verdict,
|
||||
confidence=confidence,
|
||||
s_total=s_total,
|
||||
s_company=s_company,
|
||||
s_macro=s_macro,
|
||||
s_competitive=s_competitive,
|
||||
signal_weights=signal_weights,
|
||||
reasoning=reasoning,
|
||||
)
|
||||
Reference in New Issue
Block a user