feat: implement dual-pipeline signal engine service
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled

New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.

Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
  gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading

Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant

Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)
This commit is contained in:
Celes Renata
2026-05-02 07:32:26 +00:00
parent 7e2343ec2c
commit f468e30af0
61 changed files with 14107 additions and 184 deletions
+326
View File
@@ -0,0 +1,326 @@
"""Unit tests for services.signal_engine.config.
Covers:
- Default values and fail-safe behaviour
- DB row parsing and application
- Environment variable overrides
- Sub-config derivation properties
- load_config() with mocked asyncpg pool
"""
from __future__ import annotations
import json
import os
from unittest.mock import AsyncMock, patch
import pytest
from services.signal_engine.config import (
ExitConfig,
HardFilterConfig,
HeuristicConfig,
ProbabilisticConfig,
SignalEngineConfig,
_apply_db_rows,
_apply_env_overrides,
_parse_value,
load_config,
)
# ---------------------------------------------------------------------------
# Defaults
# ---------------------------------------------------------------------------
class TestDefaults:
"""SignalEngineConfig defaults match the design spec."""
def test_dual_pipeline_disabled_by_default(self):
cfg = SignalEngineConfig()
assert cfg.dual_pipeline_enabled is False
def test_both_pipelines_enabled_by_default(self):
cfg = SignalEngineConfig()
assert cfg.heuristic_pipeline_enabled is True
assert cfg.probabilistic_pipeline_enabled is True
def test_shadow_mode_off_by_default(self):
cfg = SignalEngineConfig()
assert cfg.shadow_mode is False
def test_timeframe_weights_default(self):
cfg = SignalEngineConfig()
expected = {
"M30": 0.03,
"H1": 0.07,
"H4": 0.15,
"D": 0.30,
"W": 0.30,
"M": 0.15,
}
assert cfg.timeframe_weights == expected
def test_hard_filter_defaults(self):
cfg = SignalEngineConfig()
assert cfg.hard_filter_valuation_min == 0.3
assert cfg.hard_filter_earnings_days == 5
assert cfg.hard_filter_macro_bias_skip == -1.0
def test_heuristic_threshold_defaults(self):
cfg = SignalEngineConfig()
assert cfg.heuristic_buy_confidence == 0.70
assert cfg.heuristic_buy_s_total == 1.2
assert cfg.heuristic_buy_valuation_min == 0.5
assert cfg.heuristic_watch_confidence == 0.55
def test_probabilistic_threshold_defaults(self):
cfg = SignalEngineConfig()
assert cfg.prob_buy_p_up == 0.60
assert cfg.prob_buy_entropy_max == 0.90
assert cfg.prob_buy_ev_r_min == 1.5
assert cfg.prob_buy_valuation_min == 0.5
assert cfg.prob_watch_p_up == 0.55
assert cfg.prob_watch_entropy_max == 0.95
assert cfg.prob_entropy_skip == 0.95
def test_regime_prior_defaults(self):
cfg = SignalEngineConfig()
assert cfg.regime_prior_bull == 0.58
assert cfg.regime_prior_range == 0.50
assert cfg.regime_prior_bear == 0.42
def test_exit_and_polling_defaults(self):
cfg = SignalEngineConfig()
assert cfg.trailing_stop_atr_multiplier == 2.0
assert cfg.polling_interval_seconds == 30
# ---------------------------------------------------------------------------
# Sub-config derivation
# ---------------------------------------------------------------------------
class TestSubConfigs:
"""Properties derive correct sub-config instances."""
def test_hard_filter_config(self):
cfg = SignalEngineConfig(
hard_filter_valuation_min=0.4,
hard_filter_earnings_days=7,
hard_filter_macro_bias_skip=-0.5,
)
hf = cfg.hard_filter_config
assert isinstance(hf, HardFilterConfig)
assert hf.valuation_min == 0.4
assert hf.earnings_days == 7
assert hf.macro_bias_skip == -0.5
def test_heuristic_config(self):
cfg = SignalEngineConfig(
heuristic_buy_confidence=0.80,
heuristic_buy_s_total=1.5,
heuristic_buy_valuation_min=0.6,
heuristic_watch_confidence=0.60,
hard_filter_earnings_days=10,
)
hc = cfg.heuristic_config
assert isinstance(hc, HeuristicConfig)
assert hc.buy_confidence == 0.80
assert hc.buy_s_total == 1.5
assert hc.buy_valuation_min == 0.6
assert hc.watch_confidence == 0.60
assert hc.macro_bias_threshold == 0.0
assert hc.earnings_days_threshold == 10
def test_probabilistic_config(self):
cfg = SignalEngineConfig(
prob_buy_p_up=0.65,
regime_prior_bull=0.60,
)
pc = cfg.probabilistic_config
assert isinstance(pc, ProbabilisticConfig)
assert pc.buy_p_up == 0.65
assert pc.regime_prior_bull == 0.60
assert pc.macro_bias_threshold == 0.0
def test_exit_config(self):
cfg = SignalEngineConfig(trailing_stop_atr_multiplier=3.0)
ec = cfg.exit_config
assert isinstance(ec, ExitConfig)
assert ec.trailing_stop_atr_multiplier == 3.0
# ---------------------------------------------------------------------------
# _parse_value
# ---------------------------------------------------------------------------
class TestParseValue:
def test_bool_true_variants(self):
for v in ("true", "True", "TRUE", "1", "yes"):
assert _parse_value(v, bool) is True
def test_bool_false_variants(self):
for v in ("false", "False", "0", "no", "anything"):
assert _parse_value(v, bool) is False
def test_int(self):
assert _parse_value("42", int) == 42
def test_float(self):
assert _parse_value("0.75", float) == 0.75
def test_dict_json(self):
raw = json.dumps({"D": 0.30, "W": 0.30})
result = _parse_value(raw, dict)
assert result == {"D": 0.30, "W": 0.30}
def test_invalid_int_raises(self):
with pytest.raises(ValueError):
_parse_value("not_a_number", int)
def test_invalid_json_raises(self):
with pytest.raises(json.JSONDecodeError):
_parse_value("{bad json", dict)
# ---------------------------------------------------------------------------
# _apply_db_rows
# ---------------------------------------------------------------------------
class TestApplyDbRows:
def test_applies_known_keys(self):
cfg = SignalEngineConfig()
rows = [
("signal_engine_dual_pipeline_enabled", "true"),
("signal_engine_prob_buy_p_up", "0.65"),
("signal_engine_polling_interval_seconds", "60"),
]
_apply_db_rows(cfg, rows)
assert cfg.dual_pipeline_enabled is True
assert cfg.prob_buy_p_up == 0.65
assert cfg.polling_interval_seconds == 60
def test_ignores_unknown_keys(self):
cfg = SignalEngineConfig()
rows = [("signal_engine_unknown_field", "whatever")]
_apply_db_rows(cfg, rows) # should not raise
def test_invalid_value_keeps_default(self):
cfg = SignalEngineConfig()
rows = [("signal_engine_hard_filter_earnings_days", "not_a_number")]
_apply_db_rows(cfg, rows)
assert cfg.hard_filter_earnings_days == 5 # default preserved
def test_timeframe_weights_from_json(self):
cfg = SignalEngineConfig()
new_weights = {"D": 0.50, "W": 0.50}
rows = [
("signal_engine_timeframe_weights", json.dumps(new_weights)),
]
_apply_db_rows(cfg, rows)
assert cfg.timeframe_weights == new_weights
# ---------------------------------------------------------------------------
# _apply_env_overrides
# ---------------------------------------------------------------------------
class TestApplyEnvOverrides:
def test_env_override_bool(self):
cfg = SignalEngineConfig()
with patch.dict(os.environ, {"SIGNAL_ENGINE_DUAL_PIPELINE_ENABLED": "true"}):
_apply_env_overrides(cfg)
assert cfg.dual_pipeline_enabled is True
def test_env_override_float(self):
cfg = SignalEngineConfig()
with patch.dict(os.environ, {"SIGNAL_ENGINE_PROB_BUY_P_UP": "0.70"}):
_apply_env_overrides(cfg)
assert cfg.prob_buy_p_up == 0.70
def test_env_override_int(self):
cfg = SignalEngineConfig()
with patch.dict(os.environ, {"SIGNAL_ENGINE_POLLING_INTERVAL_SECONDS": "120"}):
_apply_env_overrides(cfg)
assert cfg.polling_interval_seconds == 120
def test_env_ignores_unrelated_vars(self):
cfg = SignalEngineConfig()
with patch.dict(os.environ, {"UNRELATED_VAR": "hello"}):
_apply_env_overrides(cfg)
# No change — just verifying no crash
assert cfg.dual_pipeline_enabled is False
def test_invalid_env_value_keeps_previous(self):
cfg = SignalEngineConfig()
cfg.hard_filter_earnings_days = 10
with patch.dict(os.environ, {"SIGNAL_ENGINE_HARD_FILTER_EARNINGS_DAYS": "bad"}):
_apply_env_overrides(cfg)
assert cfg.hard_filter_earnings_days == 10 # unchanged
# ---------------------------------------------------------------------------
# load_config (async)
# ---------------------------------------------------------------------------
class TestLoadConfig:
@pytest.mark.asyncio
async def test_load_with_db_rows(self):
"""DB rows are applied over defaults."""
pool = AsyncMock()
pool.fetch = AsyncMock(
return_value=[
{"key": "signal_engine_dual_pipeline_enabled", "value": "true"},
{"key": "signal_engine_shadow_mode", "value": "true"},
]
)
cfg = await load_config(pool)
assert cfg.dual_pipeline_enabled is True
assert cfg.shadow_mode is True
@pytest.mark.asyncio
async def test_load_with_empty_db(self):
"""Empty DB result returns safe defaults."""
pool = AsyncMock()
pool.fetch = AsyncMock(return_value=[])
cfg = await load_config(pool)
assert cfg.dual_pipeline_enabled is False
assert cfg.heuristic_pipeline_enabled is True
@pytest.mark.asyncio
async def test_load_db_failure_failsafe(self):
"""DB error falls back to disabled (fail-safe)."""
pool = AsyncMock()
pool.fetch = AsyncMock(side_effect=Exception("connection refused"))
cfg = await load_config(pool)
assert cfg.dual_pipeline_enabled is False
@pytest.mark.asyncio
async def test_env_overrides_db_values(self):
"""Environment variables take precedence over DB values."""
pool = AsyncMock()
pool.fetch = AsyncMock(
return_value=[
{"key": "signal_engine_prob_buy_p_up", "value": "0.55"},
]
)
with patch.dict(os.environ, {"SIGNAL_ENGINE_PROB_BUY_P_UP": "0.70"}):
cfg = await load_config(pool)
assert cfg.prob_buy_p_up == 0.70 # env wins
@pytest.mark.asyncio
async def test_env_overrides_applied_after_db_failure(self):
"""Env overrides still apply even when DB read fails."""
pool = AsyncMock()
pool.fetch = AsyncMock(side_effect=Exception("timeout"))
with patch.dict(
os.environ, {"SIGNAL_ENGINE_DUAL_PIPELINE_ENABLED": "true"}
):
cfg = await load_config(pool)
# Env override can re-enable even after DB failure
assert cfg.dual_pipeline_enabled is True