feat: implement dual-pipeline signal engine service
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled

New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.

Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
  gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading

Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant

Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)
This commit is contained in:
Celes Renata
2026-05-02 07:32:26 +00:00
parent 7e2343ec2c
commit f468e30af0
61 changed files with 14107 additions and 184 deletions
+148
View File
@@ -0,0 +1,148 @@
#!/usr/bin/env node
/**
* Minimal MCP server for OpenAI chat completions.
* Accepts ANY model string (gpt-5.2, gpt-5.4, etc.) — no hardcoded enum.
* Communicates over stdio using JSON-RPC (MCP protocol).
*/
import { createInterface } from "readline";
const OPENAI_API_KEY = process.env.OPENAI_API_KEY;
if (!OPENAI_API_KEY) {
process.stderr.write("ERROR: OPENAI_API_KEY environment variable is required\n");
process.exit(1);
}
const SERVER_INFO = {
name: "openai-chat",
version: "1.0.0",
};
const TOOLS = [
{
name: "openai_chat",
description:
"Send messages to OpenAI chat completions API. Supports all OpenAI models including GPT-5.x series.",
inputSchema: {
type: "object",
properties: {
model: {
type: "string",
description:
"OpenAI model name (e.g. gpt-5.2, gpt-5.4, gpt-4o, etc.)",
default: "gpt-5.2",
},
messages: {
type: "array",
description: "Array of chat messages",
items: {
type: "object",
properties: {
role: {
type: "string",
enum: ["system", "user", "assistant"],
},
content: { type: "string" },
},
required: ["role", "content"],
},
},
},
required: ["messages"],
},
},
];
async function callOpenAI(model, messages) {
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${OPENAI_API_KEY}`,
},
body: JSON.stringify({ model, messages }),
});
if (!resp.ok) {
const errText = await resp.text();
throw new Error(`OpenAI API error ${resp.status}: ${errText}`);
}
const data = await resp.json();
return data.choices?.[0]?.message?.content ?? "(no response)";
}
function jsonRpcResponse(id, result) {
return JSON.stringify({ jsonrpc: "2.0", id, result });
}
function jsonRpcError(id, code, message) {
return JSON.stringify({ jsonrpc: "2.0", id, error: { code, message } });
}
async function handleRequest(req) {
const { id, method, params } = req;
switch (method) {
case "initialize":
return jsonRpcResponse(id, {
protocolVersion: "2024-11-05",
capabilities: { tools: {} },
serverInfo: SERVER_INFO,
});
case "notifications/initialized":
return null; // no response needed for notifications
case "tools/list":
return jsonRpcResponse(id, { tools: TOOLS });
case "tools/call": {
const toolName = params?.name;
if (toolName !== "openai_chat") {
return jsonRpcError(id, -32602, `Unknown tool: ${toolName}`);
}
const args = params?.arguments ?? {};
const model = args.model || "gpt-5.2";
const messages = args.messages || [];
if (!messages.length) {
return jsonRpcError(id, -32602, "messages array is required");
}
try {
const content = await callOpenAI(model, messages);
return jsonRpcResponse(id, {
content: [{ type: "text", text: content }],
});
} catch (err) {
return jsonRpcResponse(id, {
content: [{ type: "text", text: `Error: ${err.message}` }],
isError: true,
});
}
}
case "ping":
return jsonRpcResponse(id, {});
default:
if (method?.startsWith("notifications/")) return null;
return jsonRpcError(id, -32601, `Method not found: ${method}`);
}
}
// stdio transport
const rl = createInterface({ input: process.stdin });
rl.on("line", async (line) => {
try {
const req = JSON.parse(line);
const resp = await handleRequest(req);
if (resp) {
process.stdout.write(resp + "\n");
}
} catch (err) {
process.stderr.write(`Parse error: ${err.message}\n`);
}
});
@@ -0,0 +1 @@
{"specId": "d76705a8-fb91-4fce-b59e-c4b3b0dbbd83", "workflowType": "requirements-first", "specType": "feature"}
@@ -0,0 +1,723 @@
# Design Document — Dual-Pipeline Signal Engine
## Overview
The dual-pipeline signal engine is a new service at `services/signal_engine/` that runs as an independent Kubernetes deployment alongside the existing aggregation → recommendation pipeline. It implements a concurrent dual-pipeline architecture where both a heuristic (deterministic scoring) and probabilistic (Bayesian inference) pipeline evaluate the same normalized inputs per ticker per evaluation tick, producing independent BUY/WATCH/SKIP verdicts. A delta analyzer compares the two verdicts, and an output formatter assembles a structured `SignalOutput` contract published to the existing `trading_decisions` Redis queue.
The engine introduces several new components — Input Normalizer, Signal Library (Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave), Multi-Timeframe Engine, Hard Filter Engine, Exit Engine, Delta Analyzer, and Output Formatter — while reusing existing infrastructure: `compute_signal_weight`, `compute_bayesian_posterior`, `classify_regime`, `WeightedSignal`, `BayesianPosterior`, and `RegimeClassification` from `services/aggregation/`.
The service is toggled via `dual_pipeline_enabled` in the `risk_configs` table (default: false, fail-safe). When disabled, the existing pipeline operates unchanged. When enabled, the signal engine runs alongside the existing pipeline with support for shadow mode (dual-pipeline output persisted but not forwarded to trading).
### Design Rationale
- **Separate service, not inline extension**: The signal engine has a fundamentally different evaluation cadence (multi-timeframe technical signals) and data flow (OHLCV bars, not document intelligence). Embedding it in the aggregation worker would couple two distinct concerns.
- **Reuse existing math**: The Bayesian posterior, regime classification, and signal weighting functions are battle-tested. The probabilistic pipeline wraps them with regime-based priors and likelihood ratio accumulation rather than reimplementing.
- **Concurrent pipelines via asyncio.gather**: Both pipelines share the same `NormalizedInput` reference and run concurrently. If one fails, the other completes normally with the failed pipeline producing a SKIP verdict.
- **Signal clustering for correlation penalty**: The Bayesian pipeline groups signals into four clusters (momentum, structure, volatility, fundamentals) and applies exponential decay within each cluster to prevent likelihood ratio stacking inflation from correlated signals.
---
## Architecture
### High-Level Flow
```mermaid
graph TD
A[Evaluation Tick<br/>Redis queue: signal_engine] --> B[Input Normalizer]
B --> C[Hard Filter Engine]
C -->|filtered out| D[SKIP verdict for both pipelines]
C -->|passed| E[Signal Library]
E --> F[Multi-Timeframe Engine]
F --> G{asyncio.gather}
G --> H[Heuristic Pipeline]
G --> I[Probabilistic Pipeline]
H --> J[Delta Analyzer]
I --> J
J --> K[Output Formatter]
K --> L[SignalOutput]
L --> M[Redis: trading_decisions queue]
L --> N[PostgreSQL: signal_engine_outputs]
subgraph Exit Path
B --> O[Exit Engine]
O --> K
end
```
### Trigger Mechanism
The signal engine polls a new Redis queue `stonks:queue:signal_engine`. Evaluation ticks are enqueued by the scheduler service after aggregation completes for a ticker. The queue message contains `{"ticker": "AAPL", "triggered_at": "2024-01-15T10:00:00Z"}`.
### Integration Points
| Component | Integration | Direction |
|---|---|---|
| Scheduler | Enqueues ticks to `signal_engine` queue | Scheduler → Signal Engine |
| Market data tables | OHLCV bars, closing prices, returns | Signal Engine reads |
| `macro_impact_records` | Macro bias computation | Signal Engine reads |
| `trend_windows` | Fundamental/valuation context | Signal Engine reads |
| `risk_configs` | Feature flags, thresholds | Signal Engine reads |
| `classify_regime()` | Regime classification for priors | Signal Engine calls |
| `compute_signal_weight()` | Heuristic signal weighting | Signal Engine calls |
| `compute_bayesian_posterior()` | Bayesian accumulation | Signal Engine calls |
| Redis `trading_decisions` | SignalOutput publication | Signal Engine → Trading Engine |
| `signal_engine_outputs` table | Persistence for audit | Signal Engine writes |
| Redis rolling agreement | Delta analyzer metrics | Signal Engine writes |
---
## Components and Interfaces
### Module Structure
```
services/signal_engine/
├── __init__.py
├── main.py # Entry point: asyncio event loop, queue polling
├── worker.py # Top-level orchestrator per evaluation tick
├── config.py # SignalEngineConfig, loaded from risk_configs + env
├── models.py # All Pydantic models (NormalizedInput, SignalResult, etc.)
├── normalizer.py # Input Normalizer — fetches and assembles NormalizedInput
├── signals/
│ ├── __init__.py
│ ├── base.py # SignalEvaluator protocol, SignalResult model
│ ├── fibonacci.py # Fibonacci retracement evaluator
│ ├── ma_stack.py # Moving average stack evaluator
│ ├── rsi.py # RSI evaluator
│ ├── cup_handle.py # Cup & Handle pattern detector
│ └── elliott_wave.py # Elliott Wave detector
├── confluence.py # Multi-Timeframe Confluence Engine
├── hard_filter.py # Hard Filter Engine
├── heuristic.py # Heuristic Pipeline (Pipeline A)
├── probabilistic.py # Probabilistic Pipeline (Pipeline B)
├── correlation.py # Signal cluster classification + correlation penalty
├── exit_engine.py # Exit Engine — position-level exit management
├── delta.py # Delta Analyzer
├── formatter.py # Output Formatter
└── persistence.py # Database persistence for signal_engine_outputs
```
### Key Function Signatures
#### `main.py` — Entry Point
```python
async def main() -> None:
"""Start the signal engine worker loop.
Connects to PostgreSQL and Redis, loads config from risk_configs,
and polls the signal_engine queue indefinitely.
"""
```
#### `worker.py` — Orchestrator
```python
async def evaluate_tick(
pool: asyncpg.Pool,
redis: redis.asyncio.Redis,
ticker: str,
config: SignalEngineConfig,
) -> SignalOutput | None:
"""Run a full evaluation tick for a single ticker.
1. Normalize inputs
2. Evaluate exit conditions for open positions
3. Run hard filters
4. Evaluate signals across timeframes
5. Run both pipelines concurrently
6. Compute delta analysis
7. Format and publish output
Returns None if the ticker is hard-filtered or both pipelines fail.
"""
```
#### `normalizer.py` — Input Normalizer
```python
async def normalize_input(
pool: asyncpg.Pool,
ticker: str,
config: SignalEngineConfig,
) -> NormalizedInput:
"""Fetch and assemble all data needed for a single evaluation tick.
Sources:
- OHLCV bars from market_data_bars (M30, H1, H4, D, W, M)
- Fundamental metrics from trend_windows + companies
- Macro context from macro_impact_records + global_events
- Open position state from the trading engine's portfolio
Missing data sources produce sentinel values (None/empty list)
with a logged warning.
"""
```
#### `signals/base.py` — Signal Evaluator Protocol
```python
from typing import Protocol
class SignalEvaluator(Protocol):
"""Protocol for all signal evaluators in the Signal Library."""
def evaluate(
self,
bars: list[OHLCVBar],
timeframe: str,
) -> SignalResult | None:
"""Evaluate a signal on a single timeframe's bar data.
Returns None when insufficient data is available.
"""
...
```
#### `confluence.py` — Multi-Timeframe Engine
```python
def compute_confluence(
signal_results: dict[str, dict[str, SignalResult]],
weights: dict[str, float],
) -> list[ConfluenceSignal]:
"""Compute weighted confluence scores across timeframes.
Args:
signal_results: {signal_type: {timeframe: SignalResult}}
weights: {timeframe: weight} e.g. {"M30": 0.03, "D": 0.30, ...}
Returns:
List of ConfluenceSignal objects that pass the minimum
confluence threshold (≥2 timeframes, ≥1 of D/W/M).
"""
```
#### `hard_filter.py` — Hard Filter Engine
```python
def evaluate_hard_filters(
normalized: NormalizedInput,
config: HardFilterConfig,
) -> HardFilterResult:
"""Evaluate pre-pipeline hard filters.
Checks:
- macro_bias == -1.0 → SKIP
- valuation_score < threshold → SKIP
- earnings_proximity_days <= threshold → SKIP
Returns HardFilterResult with filtered=True/False and all triggered reasons.
"""
```
#### `heuristic.py` — Heuristic Pipeline
```python
def run_heuristic_pipeline(
normalized: NormalizedInput,
confluence_signals: list[ConfluenceSignal],
config: HeuristicConfig,
) -> HeuristicResult:
"""Run the deterministic heuristic pipeline.
Computes S_total = S_company + S_macro + S_competitive using
existing compute_signal_weight() and weighted sentiment averaging.
Produces BUY/WATCH/SKIP verdict based on confidence and score thresholds.
"""
```
#### `probabilistic.py` — Probabilistic Pipeline
```python
def run_probabilistic_pipeline(
normalized: NormalizedInput,
confluence_signals: list[ConfluenceSignal],
regime: RegimeClassification,
config: ProbabilisticConfig,
) -> ProbabilisticResult:
"""Run the Bayesian probabilistic pipeline.
1. Initialize regime-based prior (bull=0.58, range=0.50, bear=0.42)
2. Compute likelihood ratios per signal with correlation penalty
3. Accumulate via log-odds: logit(P_post) = logit(P_prior) + Σ log(LR_i)
4. Apply entropy gating
5. Compute EV_R = P_up · E[win_R] - (1 - P_up) · 1.0
6. Produce BUY/WATCH/SKIP verdict
"""
```
#### `correlation.py` — Signal Correlation Penalty
```python
class SignalCluster(str, Enum):
MOMENTUM = "momentum" # MA stack, RSI
STRUCTURE = "structure" # Fibonacci, Elliott Wave
VOLATILITY = "volatility" # ATR-based, Bollinger-derived
FUNDAMENTALS = "fundamentals" # valuation, earnings, macro
def classify_signal(signal_type: str) -> SignalCluster:
"""Map a signal type to its correlation cluster."""
def apply_correlation_penalty(
likelihood_ratios: list[LikelihoodRatio],
) -> list[LikelihoodRatio]:
"""Apply within-cluster decay penalty to correlated signals.
Within each cluster, signals are ranked by LR magnitude.
The strongest contributes at full weight; subsequent signals
contribute at 0.5^(n-1) decay.
Cross-cluster signals are independent (no penalty).
"""
```
#### `exit_engine.py` — Exit Engine
```python
def evaluate_exits(
positions: list[OpenPositionState],
current_prices: dict[str, float],
config: ExitConfig,
) -> list[ExitSignal]:
"""Evaluate exit conditions for all open positions.
Checks: stop_loss hit, target_1 hit (EXIT_HALF), target_2 hit (EXIT_FULL),
trailing stop hit (EXIT_FULL for remaining).
Trailing stop activates after EXIT_HALF and ratchets upward only.
"""
```
#### `delta.py` — Delta Analyzer
```python
async def analyze_delta(
heuristic: HeuristicResult,
probabilistic: ProbabilisticResult,
redis: redis.asyncio.Redis,
ticker: str,
) -> DeltaResult:
"""Compare pipeline verdicts and track agreement metrics.
Computes agreement flag, confidence delta, disagreement reasons.
Updates rolling 100-evaluation agreement rate in Redis.
Logs warning when agreement rate drops below 0.50.
"""
```
#### `formatter.py` — Output Formatter
```python
def format_output(
ticker: str,
price: float,
heuristic: HeuristicResult,
probabilistic: ProbabilisticResult,
delta: DeltaResult,
exit_signals: list[ExitSignal],
config: SignalEngineConfig,
) -> SignalOutput:
"""Assemble the structured SignalOutput contract.
Populates trade_plan based on verdict combination:
- Both BUY → dual_confirmed, full position sizing
- Probabilistic-only BUY → probabilistic_only, 50% position sizing
- Heuristic-only BUY → standard position sizing
- No BUY → no trade_plan (WATCH/SKIP persisted for analysis)
"""
def signal_output_to_recommendation(output: SignalOutput) -> Recommendation:
"""Map a SignalOutput to the existing Recommendation schema.
Enables the trading engine to consume dual-pipeline outputs
without modification to its core evaluate_recommendation logic.
"""
```
#### `persistence.py` — Database Persistence
```python
async def persist_signal_output(
pool: asyncpg.Pool,
output: SignalOutput,
) -> None:
"""Persist a SignalOutput to the signal_engine_outputs table.
Logs and continues on database errors (persistence failure
does not block signal emission to the trading queue).
"""
```
---
## Data Models
All new data models are Pydantic `BaseModel` subclasses defined in `services/signal_engine/models.py`. Existing models (`WeightedSignal`, `BayesianPosterior`, `RegimeClassification`, `TrendSummary`, `Recommendation`, `PositionSizing`) are imported from `services/aggregation/` and `services/shared/schemas.py`.
### OHLCVBar
```python
class OHLCVBar(BaseModel):
"""Single OHLCV bar for a timeframe."""
timestamp: datetime
open: float
high: float
low: float
close: float
volume: float
```
### NormalizedInput
```python
class NormalizedInput(BaseModel):
"""Unified input structure consumed by both pipelines."""
ticker: str
evaluated_at: datetime
# Multi-timeframe OHLCV bars
bars: dict[str, list[OHLCVBar]] # {"M30": [...], "H1": [...], ...}
# Fundamental metrics
valuation_score: float | None = None # [0.0, 1.0]
earnings_proximity_days: int | None = None
# Macro context
macro_bias: float = 0.0 # [-1.0, 1.0]
# Open position state (for exit engine)
open_positions: list[OpenPositionState] = Field(default_factory=list)
# Market data for regime classification
closing_prices: list[float] = Field(default_factory=list)
returns: list[float] = Field(default_factory=list)
# Current price (latest close from shortest available timeframe)
current_price: float | None = None
```
### OpenPositionState
```python
class OpenPositionState(BaseModel):
"""Snapshot of an open position for exit evaluation."""
position_id: str
ticker: str
entry_price: float
current_price: float
stop_loss: float
target_1: float
target_2: float
trailing_stop: float | None = None
partial_exit_done: bool = False
atr: float | None = None
```
### SignalResult
```python
class SignalDirection(str, Enum):
BULLISH = "bullish"
BEARISH = "bearish"
NEUTRAL = "neutral"
class SignalResult(BaseModel):
"""Output from a single signal evaluator on a single timeframe."""
signal_type: str # e.g. "fibonacci", "ma_stack", "rsi"
timeframe: str # e.g. "D", "H4"
strength: float = Field(ge=0.0, le=1.0)
direction: SignalDirection
confidence: float = Field(ge=0.0, le=1.0)
metadata: dict = Field(default_factory=dict) # signal-specific details
```
### ConfluenceSignal
```python
class ConfluenceSignal(BaseModel):
"""A signal that passed multi-timeframe confluence filtering."""
signal_type: str
direction: SignalDirection
confluence_score: float # weighted sum across timeframes
active_timeframes: list[str] # which timeframes triggered
per_timeframe: dict[str, float] # {timeframe: strength}
```
### Verdict
```python
class Verdict(str, Enum):
BUY = "BUY"
WATCH = "WATCH"
SKIP = "SKIP"
```
### HeuristicResult
```python
class HeuristicResult(BaseModel):
"""Output from the heuristic (deterministic) pipeline."""
verdict: Verdict
confidence: float = Field(ge=0.0, le=1.0)
s_total: float
s_company: float
s_macro: float
s_competitive: float
signal_weights: list[dict] = Field(default_factory=list)
reasoning: list[str] = Field(default_factory=list)
```
### LikelihoodRatio
```python
class LikelihoodRatio(BaseModel):
"""A single signal's likelihood ratio for Bayesian updating."""
signal_type: str
cluster: str # SignalCluster value
lr: float # P(sig|up) / P(sig|down)
log_lr: float # log(lr)
penalized_log_lr: float # after correlation penalty
hit_rate: float
strength: float
```
### ProbabilisticResult
```python
class ProbabilisticResult(BaseModel):
"""Output from the probabilistic (Bayesian) pipeline."""
verdict: Verdict
p_up: float = Field(ge=0.0, le=1.0)
entropy: float = Field(ge=0.0, le=1.0)
ev_r: float
prior: float
posterior: float
likelihood_ratios: list[LikelihoodRatio] = Field(default_factory=list)
regime: str
reasoning: list[str] = Field(default_factory=list)
```
### DeltaResult
```python
class DeltaResult(BaseModel):
"""Output from the delta analyzer comparing both pipelines."""
agreement: bool
confidence_delta: float
heuristic_verdict: str
probabilistic_verdict: str
disagreement_reasons: list[str] = Field(default_factory=list)
rolling_agreement_rate: float | None = None
```
### ExitSignal
```python
class ExitType(str, Enum):
EXIT_HALF = "EXIT_HALF"
EXIT_FULL = "EXIT_FULL"
class ExitSignal(BaseModel):
"""An exit signal for an open position."""
position_id: str
ticker: str
exit_type: ExitType
reason: str # "stop_hit", "target_1_hit", "target_2_hit", "trailing_stop_hit"
price: float
```
### TradePlan
```python
class TradePlan(BaseModel):
"""Optional trade plan attached to a BUY signal."""
entry_price: float
stop_loss: float
target_1: float
target_2: float
position_size_pct: float = Field(ge=0.0, le=1.0)
max_loss_pct: float = Field(ge=0.0, le=1.0)
dual_confirmed: bool = False
probabilistic_only: bool = False
```
### SignalOutput
```python
class SignalOutput(BaseModel):
"""The structured output contract consumed by the trading engine and audit systems."""
output_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
ticker: str
timestamp: datetime
price: float
# Heuristic pipeline results
heuristic_verdict: str
heuristic_confidence: float
heuristic_s_total: float
# Probabilistic pipeline results
probabilistic_verdict: str
probabilistic_p_up: float
probabilistic_entropy: float
probabilistic_ev_r: float
# Delta analysis
delta_agreement: bool
delta_confidence_delta: float
delta_reasons: list[str] = Field(default_factory=list)
# Optional trade plan (populated when at least one pipeline says BUY)
trade_plan: TradePlan | None = None
# Exit signals for open positions
exit_signals: list[ExitSignal] = Field(default_factory=list)
# Full pipeline results for audit (stored as JSONB)
heuristic_detail: dict = Field(default_factory=dict)
probabilistic_detail: dict = Field(default_factory=dict)
# Pipeline mode metadata
pipeline_mode: str = "dual_pipeline"
shadow_mode: bool = False
```
### SignalEngineConfig
```python
@dataclass
class SignalEngineConfig:
"""Configuration loaded from risk_configs + environment."""
dual_pipeline_enabled: bool = False
heuristic_pipeline_enabled: bool = True
probabilistic_pipeline_enabled: bool = True
shadow_mode: bool = False
# Timeframe weights
timeframe_weights: dict[str, float] = field(default_factory=lambda: {
"M30": 0.03, "H1": 0.07, "H4": 0.15,
"D": 0.30, "W": 0.30, "M": 0.15,
})
# Hard filter thresholds
hard_filter_valuation_min: float = 0.3
hard_filter_earnings_days: int = 5
hard_filter_macro_bias_skip: float = -1.0
# Heuristic verdict thresholds
heuristic_buy_confidence: float = 0.70
heuristic_buy_s_total: float = 1.2
heuristic_buy_valuation_min: float = 0.5
heuristic_watch_confidence: float = 0.55
# Probabilistic verdict thresholds
prob_buy_p_up: float = 0.60
prob_buy_entropy_max: float = 0.90
prob_buy_ev_r_min: float = 1.5
prob_buy_valuation_min: float = 0.5
prob_watch_p_up: float = 0.55
prob_watch_entropy_max: float = 0.95
prob_entropy_skip: float = 0.95
# Regime priors
regime_prior_bull: float = 0.58
regime_prior_range: float = 0.50
regime_prior_bear: float = 0.42
# Exit engine
trailing_stop_atr_multiplier: float = 2.0
# Polling
polling_interval_seconds: int = 30
```
### HardFilterConfig / HeuristicConfig / ProbabilisticConfig / ExitConfig
These are derived from `SignalEngineConfig` fields for cleaner function signatures — simple `@dataclass` wrappers over the relevant subset of config values.
---
### Database Migration (039)
```sql
-- Migration 039: Signal Engine Outputs
-- Creates the signal_engine_outputs table for persisting dual-pipeline evaluations.
CREATE TABLE IF NOT EXISTS signal_engine_outputs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
ticker TEXT NOT NULL,
evaluated_at TIMESTAMPTZ NOT NULL,
price NUMERIC NOT NULL,
-- Heuristic pipeline
heuristic_verdict TEXT NOT NULL,
heuristic_confidence NUMERIC NOT NULL,
heuristic_s_total NUMERIC NOT NULL,
-- Probabilistic pipeline
probabilistic_verdict TEXT NOT NULL,
probabilistic_p_up NUMERIC NOT NULL,
probabilistic_entropy NUMERIC NOT NULL,
probabilistic_ev_r NUMERIC NOT NULL,
-- Delta analysis
delta_agreement BOOLEAN NOT NULL,
delta_confidence_delta NUMERIC NOT NULL,
delta_reasons JSONB NOT NULL DEFAULT '[]'::jsonb,
-- Trade plan (null when no BUY verdict)
trade_plan JSONB,
-- Full output for audit
full_output JSONB NOT NULL,
-- Exit signals
exit_signals JSONB NOT NULL DEFAULT '[]'::jsonb,
-- Metadata
pipeline_mode TEXT NOT NULL DEFAULT 'dual_pipeline',
shadow_mode BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Index for per-ticker time-range queries
CREATE INDEX IF NOT EXISTS idx_signal_engine_outputs_ticker_time
ON signal_engine_outputs (ticker, evaluated_at);
-- Index for global time-range queries
CREATE INDEX IF NOT EXISTS idx_signal_engine_outputs_evaluated
ON signal_engine_outputs (evaluated_at);
-- Index for filtering by verdict
CREATE INDEX IF NOT EXISTS idx_signal_engine_outputs_verdicts
ON signal_engine_outputs (heuristic_verdict, probabilistic_verdict);
```
### Helm / Deployment Configuration
Add to `values.yaml` under `services:`:
```yaml
signalEngine:
replicas: 1
pipeline: true
image: signal-engine
command: "python -m services.signal_engine.main"
tier: processing
secrets: [stonks-core-secrets, stonks-market-secrets]
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: 500m, memory: 256Mi }
```
Add to `redis_keys.py`:
```python
QUEUE_SIGNAL_ENGINE = "signal_engine"
```
The service uses the existing `stonks-config` ConfigMap and `stonks-core-secrets` for database/Redis credentials. No new ingress or network policy is needed — the signal engine is a queue-polling worker with no HTTP interface.
---
@@ -0,0 +1,300 @@
# Requirements Document — Dual-Pipeline Signal Engine
## Introduction
The Stonks Oracle platform currently operates a single aggregation pipeline that can run in either heuristic or probabilistic mode (toggled via `probabilistic_scoring_enabled`). This feature replaces the single-pipeline toggle with a dual-pipeline architecture where both pipelines run concurrently per evaluation tick, produce independent verdicts (BUY/WATCH/SKIP), and emit a structured output contract for downstream consumers (trading engine, delta analysis, dashboards).
The dual-pipeline engine introduces:
- **Pipeline A (Heuristic)**: Deterministic scoring using the existing `S_total = S_company + S_macro + S_competitive` formula with signal weighting, producing a confidence-gated verdict.
- **Pipeline B (Probabilistic)**: Bayesian inference using the existing `bayesian.py` infrastructure with regime-based priors, likelihood ratios, entropy gating, and expected value calculation.
- **Hard Filter Engine**: Pre-pipeline filters that short-circuit both pipelines before evaluation.
- **Multi-Timeframe Engine**: Signal evaluation across M30, H1, H4, D, W, M timeframes with weighted confluence scoring.
- **Exit Engine**: Position-level exit management (stop hit, targets, trailing ATR-based).
- **Delta Analyzer**: Compares heuristic vs probabilistic verdicts to generate training signals for future model tuning.
- **Output Formatter**: Structured `SignalOutput` contract consumed by the trading engine and delta analysis.
The design must address the signal independence assumption in the Bayesian pipeline — correlated signals (MA+RSI, Fib+Elliott) require correlation penalty or signal clustering into categories (momentum, structure, volatility, fundamentals) to prevent likelihood ratio stacking inflation.
## Glossary
- **Signal_Engine**: The top-level orchestrator in `services/signal_engine/` that coordinates input normalization, hard filters, both pipelines, delta analysis, and output formatting per evaluation tick.
- **Heuristic_Pipeline**: Pipeline A — deterministic scoring that computes `S_total = S_company + S_macro + S_competitive` with signal weighting and produces a confidence-gated BUY/WATCH/SKIP verdict.
- **Probabilistic_Pipeline**: Pipeline B — Bayesian inference pipeline that computes posterior probability via log-likelihood accumulation with regime-based priors, entropy gating, and expected value calculation.
- **Input_Normalizer**: The component that ingests multi-timeframe OHLCV data, fundamentals, macro context, and open positions into a unified `NormalizedInput` structure consumed by both pipelines.
- **Signal_Library**: The collection of technical signal evaluators (Fibonacci retracement, MA stack, RSI, Cup & Handle, Elliott Wave) that produce scored signals per timeframe.
- **Multi_Timeframe_Engine**: The component that evaluates signals across six timeframes (M30, H1, H4, D, W, M) and computes weighted confluence scores.
- **Hard_Filter_Engine**: The pre-pipeline filter stage that evaluates macro bias, valuation score, and earnings proximity to short-circuit evaluation before either pipeline runs.
- **Exit_Engine**: The position management component that evaluates stop hits, take-profit targets, and trailing ATR-based stops for open positions.
- **Delta_Analyzer**: The component that compares heuristic and probabilistic verdicts, tracks agreement rates, measures confidence deltas, and records disagreement reasons as training signals.
- **Output_Formatter**: The component that assembles the structured `SignalOutput` contract from both pipeline results, delta analysis, and optional trade plan.
- **SignalOutput**: The structured output contract containing ticker, timestamp, price, heuristic verdict/confidence/S_total, probabilistic verdict/P_up/entropy/EV_R, delta analysis, and optional trade plan.
- **Verdict**: A pipeline decision of BUY, WATCH, or SKIP with associated confidence and reasoning.
- **Confluence**: The condition where a signal triggers across multiple timeframes; requires activation on at least 2 timeframes including at least one of D, W, or M.
- **Entropy_Gate**: Shannon entropy threshold used in the probabilistic pipeline to detect high-uncertainty states and force SKIP verdicts.
- **EV_R**: Expected value per unit of risk, computed as `P_up · E[win_R] - (1 - P_up) · 1.0`, used as a quality gate in the probabilistic pipeline.
- **Signal_Cluster**: A grouping of correlated signals (momentum, structure, volatility, fundamentals) used to prevent likelihood ratio stacking inflation in the Bayesian pipeline.
- **Likelihood_Ratio**: The ratio `P(signal|up) / P(signal|down)` used in Bayesian updating, where `P(sig|up) = h·s + (1-h)·(1-s)·0.5`.
- **Regime_Prior**: The initial probability assigned based on market regime classification: bull=0.58, range=0.50, bear=0.42.
- **OHLCV**: Open, High, Low, Close, Volume — standard market data bar format.
- **ATR**: Average True Range — a volatility measure used for trailing stop calculations.
- **Fibonacci_Retracement**: A technical analysis tool computing price levels as `L(r) = SH - r·(SH - SL)` where SH is swing high, SL is swing low, and r is a retracement ratio (0.236, 0.382, 0.5, 0.618, 0.786).
---
## Requirements
### Requirement 1: Input Normalization
**User Story:** As a signal engine operator, I want all market data, fundamentals, macro context, and open positions normalized into a single input structure, so that both pipelines consume identical inputs per evaluation tick.
#### Acceptance Criteria
1. WHEN an evaluation tick is triggered for a ticker, THE Input_Normalizer SHALL construct a `NormalizedInput` containing multi-timeframe OHLCV bars (M30, H1, H4, D, W, M), fundamental metrics (valuation_score, earnings_proximity_days), macro context (macro_bias as float in [-1.0, 1.0]), and open position state (entry_price, current_price, stop_loss, targets).
2. THE Input_Normalizer SHALL source OHLCV data from the existing market data tables, fundamental metrics from the existing company and trend data, and macro context from the existing `macro_impact_records` and `global_events` tables.
3. IF any required data source is unavailable or returns an error, THEN THE Input_Normalizer SHALL populate the corresponding field with a sentinel value (`None` for optional fields, empty list for OHLCV bars) and log a warning identifying the missing source.
4. THE Input_Normalizer SHALL validate that all OHLCV bars have monotonically increasing timestamps within each timeframe series.
5. THE Input_Normalizer SHALL produce identical `NormalizedInput` instances for both pipelines within the same evaluation tick (shared reference, no independent fetches).
---
### Requirement 2: Signal Library — Technical Signal Evaluation
**User Story:** As a quantitative analyst, I want a library of technical signal evaluators that produce scored signals per timeframe, so that both pipelines can consume standardized signal assessments.
#### Acceptance Criteria
1. THE Signal_Library SHALL implement Fibonacci retracement signal evaluation using the formula `L(r) = SH - r·(SH - SL)` for retracement ratios [0.236, 0.382, 0.5, 0.618, 0.786], where SH is the swing high and SL is the swing low within the evaluation window.
2. THE Signal_Library SHALL implement moving average stack evaluation that detects bullish alignment (MA_10 > MA_20 > MA_50 > MA_200) and bearish alignment (MA_10 < MA_20 < MA_50 < MA_200), producing a signal strength proportional to the degree of alignment.
3. THE Signal_Library SHALL implement RSI evaluation using the standard 14-period RSI formula, producing overbought signals (RSI > 70) and oversold signals (RSI < 30) with strength scaled by distance from the threshold.
4. THE Signal_Library SHALL implement Cup & Handle pattern detection that identifies the cup formation (U-shaped price recovery) and handle (small consolidation), producing a signal with confidence proportional to pattern completeness.
5. THE Signal_Library SHALL implement Elliott Wave detection that identifies impulse waves (5-wave structure) and corrective waves (3-wave structure), producing a signal with the current wave position and projected direction.
6. WHEN a signal evaluator receives insufficient data for its calculation (fewer bars than the required lookback period), THE Signal_Library SHALL return a null signal with a reason code indicating insufficient data rather than producing a partial evaluation.
7. FOR ALL signal evaluators, THE Signal_Library SHALL produce output conforming to a common `SignalResult` structure containing: signal_type, timeframe, strength (float in [0.0, 1.0]), direction (bullish/bearish/neutral), confidence (float in [0.0, 1.0]), and metadata specific to the signal type.
---
### Requirement 3: Multi-Timeframe Confluence Engine
**User Story:** As a quantitative analyst, I want signals evaluated across multiple timeframes with weighted confluence scoring, so that the engine prioritizes signals confirmed across longer timeframes.
#### Acceptance Criteria
1. THE Multi_Timeframe_Engine SHALL evaluate each signal type across six timeframes with the following weights: M30=0.03, H1=0.07, H4=0.15, D=0.30, W=0.30, M=0.15.
2. THE Multi_Timeframe_Engine SHALL compute a weighted confluence score as `C_confluence = Σ(w_tf · s_tf)` where `w_tf` is the timeframe weight and `s_tf` is the signal strength on that timeframe (0.0 if the signal did not trigger).
3. WHEN a signal triggers on fewer than 2 timeframes, THE Multi_Timeframe_Engine SHALL discard the signal from further pipeline processing (minimum confluence threshold).
4. WHEN a signal triggers on 2 or more timeframes but none of D, W, or M are included, THE Multi_Timeframe_Engine SHALL discard the signal from further pipeline processing (higher-timeframe anchor requirement).
5. THE Multi_Timeframe_Engine SHALL pass the confluence-filtered signals and their weighted scores to both the Heuristic_Pipeline and Probabilistic_Pipeline.
6. FOR ALL signal sets where a signal triggers on more timeframes with higher weights, THE Multi_Timeframe_Engine SHALL produce a higher confluence score (monotonicity with respect to timeframe activation count and weight).
---
### Requirement 4: Hard Filter Engine — Pre-Pipeline Gating
**User Story:** As a risk manager, I want hard filters that short-circuit both pipelines before evaluation, so that clearly unfavorable conditions produce immediate SKIP verdicts without wasting computation.
#### Acceptance Criteria
1. WHEN the macro_bias value from the NormalizedInput equals -1.0, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "macro_bias_negative".
2. WHEN the valuation_score from the NormalizedInput is below 0.3, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "valuation_below_threshold".
3. WHEN the earnings_proximity_days from the NormalizedInput is 5 or fewer, THE Hard_Filter_Engine SHALL produce an immediate SKIP verdict for both pipelines with reason "earnings_block".
4. WHEN multiple hard filters trigger simultaneously, THE Hard_Filter_Engine SHALL record all triggered filter reasons in the SKIP verdict (not just the first).
5. WHEN no hard filters trigger, THE Hard_Filter_Engine SHALL pass the NormalizedInput through to both pipelines without modification.
6. THE Hard_Filter_Engine SHALL execute before either pipeline begins evaluation, and both pipelines SHALL receive the same filter decision.
---
### Requirement 5: Heuristic Pipeline — Deterministic Scoring and Verdict
**User Story:** As a quantitative analyst, I want the heuristic pipeline to produce a deterministic BUY/WATCH/SKIP verdict based on composite scoring of company, macro, and competitive signals, so that the system maintains a transparent, auditable scoring path.
#### Acceptance Criteria
1. THE Heuristic_Pipeline SHALL compute a total score `S_total = S_company + S_macro + S_competitive` using the existing three-layer signal aggregation with the current `WeightedSignal` abstraction.
2. THE Heuristic_Pipeline SHALL compute signal weights using the formula `W_signal = gate · recency · credibility · (1 + novelty) · market_context` consistent with the existing `compute_signal_weight` function in `scoring.py`.
3. THE Heuristic_Pipeline SHALL compute a confidence value from the existing trend confidence formula incorporating source count, extraction confidence, signal agreement, and contradiction penalty.
4. THE Heuristic_Pipeline SHALL produce a BUY verdict WHEN confidence >= 0.70 AND S_total >= 1.2 AND valuation_score >= 0.5 AND macro_bias > 0 AND earnings_proximity_days > 5.
5. THE Heuristic_Pipeline SHALL produce a WATCH verdict WHEN confidence >= 0.55 AND the BUY conditions are not fully met.
6. THE Heuristic_Pipeline SHALL produce a SKIP verdict WHEN confidence < 0.55.
7. THE Heuristic_Pipeline SHALL emit a `HeuristicResult` containing: verdict (BUY/WATCH/SKIP), confidence (float), S_total (float), S_company (float), S_macro (float), S_competitive (float), signal_weights (list), and reasoning (list of strings explaining the verdict).
---
### Requirement 6: Probabilistic Pipeline — Bayesian Inference and Verdict
**User Story:** As a quantitative analyst, I want the probabilistic pipeline to produce a Bayesian BUY/WATCH/SKIP verdict using regime-based priors, likelihood ratios, entropy gating, and expected value calculation, so that the system captures uncertainty structure and risk-adjusted expected outcomes.
#### Acceptance Criteria
1. THE Probabilistic_Pipeline SHALL initialize the prior probability based on the current market regime classification: bull regime → P_prior = 0.58, range regime → P_prior = 0.50, bear regime → P_prior = 0.42.
2. THE Probabilistic_Pipeline SHALL compute likelihood ratios for each signal using `P(sig|up) = h·s + (1-h)·(1-s)·0.5` and `LR = P(sig|up) / P(sig|down)`, where h is the signal's historical hit rate and s is the signal strength.
3. THE Probabilistic_Pipeline SHALL update the posterior using log-odds accumulation: `logit(P_post) = logit(P_prior) + Σ log(LR_i)`, converting back to probability via the sigmoid function.
4. THE Probabilistic_Pipeline SHALL compute Shannon entropy `H = -P_up·log₂(P_up) - (1-P_up)·log₂(1-P_up)` and apply entropy gating: WHEN H > 0.95, THE Probabilistic_Pipeline SHALL force a SKIP verdict with reason "high_entropy".
5. THE Probabilistic_Pipeline SHALL compute expected value per unit risk as `EV_R = P_up · E[win_R] - (1 - P_up) · 1.0` where `E[win_R]` is the expected win in risk units derived from signal strength and historical reward-risk ratios.
6. THE Probabilistic_Pipeline SHALL produce a BUY verdict WHEN P_up >= 0.60 AND entropy <= 0.90 AND EV_R >= 1.5 AND macro_bias > 0 AND valuation_score >= 0.5.
7. THE Probabilistic_Pipeline SHALL produce a WATCH verdict WHEN P_up >= 0.55 AND entropy <= 0.95 AND the BUY conditions are not fully met.
8. THE Probabilistic_Pipeline SHALL produce a SKIP verdict in all other cases.
9. THE Probabilistic_Pipeline SHALL emit a `ProbabilisticResult` containing: verdict (BUY/WATCH/SKIP), P_up (float), entropy (float), EV_R (float), prior (float), posterior (float), likelihood_ratios (list), regime (string), and reasoning (list of strings).
---
### Requirement 7: Signal Correlation Penalty — Preventing LR Stacking Inflation
**User Story:** As a quantitative analyst, I want correlated signals grouped into clusters with a correlation penalty applied to prevent likelihood ratio stacking inflation, so that the Bayesian pipeline does not overstate confidence from redundant signals.
#### Acceptance Criteria
1. THE Probabilistic_Pipeline SHALL classify each signal into one of four clusters: momentum (MA stack, RSI), structure (Fibonacci retracement, Elliott Wave), volatility (ATR-based signals, Bollinger-derived), and fundamentals (valuation, earnings, macro).
2. WHEN multiple signals within the same cluster produce likelihood ratios in the same direction, THE Probabilistic_Pipeline SHALL apply a within-cluster penalty: only the strongest LR in the cluster contributes at full weight, and subsequent LRs in the same cluster contribute at a decay factor of 0.5^(n-1) where n is the signal's rank within the cluster by LR magnitude.
3. THE Probabilistic_Pipeline SHALL apply no penalty across different clusters (signals from different clusters are treated as independent).
4. WHEN a cluster contains only one signal, THE Probabilistic_Pipeline SHALL apply no penalty to that signal.
5. FOR ALL signal sets, THE Probabilistic_Pipeline SHALL produce a posterior probability that is less than or equal to the posterior computed without the correlation penalty (the penalty only reduces confidence, never inflates it).
---
### Requirement 8: Exit Engine — Position Management
**User Story:** As a trader, I want the signal engine to evaluate exit conditions for open positions, so that stop hits, take-profit targets, and trailing stops are managed as part of the signal evaluation cycle.
#### Acceptance Criteria
1. WHEN the current price of an open position hits or crosses below the stop_loss level, THE Exit_Engine SHALL emit an EXIT_FULL signal for that position with reason "stop_hit".
2. WHEN the current price of an open position hits or crosses above the first take-profit target (target_1), THE Exit_Engine SHALL emit an EXIT_HALF signal for that position with reason "target_1_hit".
3. WHEN the current price of an open position hits or crosses above the second take-profit target (target_2), THE Exit_Engine SHALL emit an EXIT_FULL signal for that position with reason "target_2_hit".
4. WHEN a partial exit has been executed (EXIT_HALF), THE Exit_Engine SHALL activate a trailing stop at `current_price - ATR · trailing_multiplier` and update the trailing stop upward as the price advances (the trailing stop moves up but does not move down).
5. WHEN the trailing stop is active and the current price crosses below the trailing stop level, THE Exit_Engine SHALL emit an EXIT_FULL signal for the remaining position with reason "trailing_stop_hit".
6. THE Exit_Engine SHALL evaluate exit conditions before the signal pipelines run for new entry signals, so that exit signals take priority over new entry signals for the same ticker.
7. THE Exit_Engine SHALL emit exit signals as part of the `SignalOutput` contract with the position identifier, exit type (EXIT_HALF/EXIT_FULL), and reason.
---
### Requirement 9: Delta Analyzer — Pipeline Agreement Tracking
**User Story:** As a model developer, I want the delta analyzer to compare heuristic and probabilistic verdicts and record disagreement details, so that I can generate training signals for future model tuning.
#### Acceptance Criteria
1. WHEN both pipelines produce verdicts for the same ticker and tick, THE Delta_Analyzer SHALL compute an agreement flag (true if both verdicts are identical, false otherwise).
2. THE Delta_Analyzer SHALL compute a confidence delta as `|heuristic_confidence - probabilistic_P_up|` representing the magnitude of disagreement between the two pipelines.
3. WHEN the pipelines disagree on verdict, THE Delta_Analyzer SHALL record the disagreement reason by identifying which conditions differed (e.g., "heuristic_confidence_below_threshold", "probabilistic_entropy_too_high", "EV_R_below_threshold").
4. THE Delta_Analyzer SHALL track a rolling agreement rate over the last 100 evaluations per ticker, stored in Redis for dashboard consumption.
5. THE Delta_Analyzer SHALL emit a `DeltaResult` containing: agreement (bool), confidence_delta (float), heuristic_verdict (string), probabilistic_verdict (string), disagreement_reasons (list of strings), and rolling_agreement_rate (float).
6. WHEN the rolling agreement rate drops below 0.50 for a ticker, THE Delta_Analyzer SHALL log a warning indicating persistent pipeline disagreement for operator review.
---
### Requirement 10: Output Formatter — Structured SignalOutput Contract
**User Story:** As a downstream system consumer, I want the signal engine to emit a structured `SignalOutput` contract, so that the trading engine, delta analysis dashboard, and audit systems can consume a consistent output format.
#### Acceptance Criteria
1. THE Output_Formatter SHALL produce a `SignalOutput` containing: ticker (string), timestamp (datetime), price (float), heuristic section (verdict, confidence, S_total), probabilistic section (verdict, P_up, entropy, EV_R), delta section (agreement, confidence_delta, disagreement_reasons), and optional trade_plan section.
2. WHEN the heuristic pipeline produces a BUY verdict, THE Output_Formatter SHALL populate the trade_plan section with entry_price, stop_loss, target_1, target_2, and position_size derived from the heuristic confidence and existing position sizing logic.
3. WHEN the probabilistic pipeline produces a BUY verdict but the heuristic pipeline does not, THE Output_Formatter SHALL populate the trade_plan section with a "probabilistic_only" flag and reduced position sizing (50% of standard).
4. WHEN both pipelines produce a BUY verdict, THE Output_Formatter SHALL populate the trade_plan section with full position sizing and a "dual_confirmed" flag.
5. THE Output_Formatter SHALL serialize the `SignalOutput` as a Pydantic model with JSON serialization support for Redis queue publishing and database persistence.
6. FOR ALL valid pipeline results, THE Output_Formatter SHALL produce a `SignalOutput` that round-trips through JSON serialization and deserialization without data loss (parse(format(output)) produces an equivalent object).
---
### Requirement 11: Dual Pipeline Orchestration
**User Story:** As a signal engine operator, I want both pipelines to run concurrently per evaluation tick sharing the same inputs, so that the system produces independent verdicts without redundant data fetching.
#### Acceptance Criteria
1. WHEN an evaluation tick is triggered, THE Signal_Engine SHALL execute the Input_Normalizer once, then pass the resulting `NormalizedInput` to the Hard_Filter_Engine, then (if not filtered) execute both the Heuristic_Pipeline and Probabilistic_Pipeline concurrently using `asyncio.gather`.
2. THE Signal_Engine SHALL enforce that both pipelines receive identical `NormalizedInput` references (no independent data fetches that could produce different snapshots).
3. WHEN either pipeline raises an exception during evaluation, THE Signal_Engine SHALL catch the exception, log the error with full traceback, and produce a SKIP verdict for the failed pipeline with reason "pipeline_error" while allowing the other pipeline to complete normally.
4. THE Signal_Engine SHALL measure and log the wall-clock execution time of each pipeline per tick for performance monitoring.
5. THE Signal_Engine SHALL publish the assembled `SignalOutput` to the existing Redis queue (`stonks:queue:trading_decisions`) for consumption by the trading engine.
6. THE Signal_Engine SHALL persist each `SignalOutput` to a database table for historical analysis and audit.
---
### Requirement 12: Integration with Existing Trading Engine
**User Story:** As a platform operator, I want the dual-pipeline signal engine to integrate with the existing trading engine, so that the trading engine can consume `SignalOutput` verdicts and make execution decisions.
#### Acceptance Criteria
1. THE Signal_Engine SHALL publish `SignalOutput` to the existing `stonks:queue:trading_decisions` Redis queue in a format compatible with the existing `TradingEngine.evaluate_recommendation` interface.
2. THE Signal_Engine SHALL map the `SignalOutput` trade_plan to the existing `Recommendation` schema fields (action, confidence, position_sizing) so that the trading engine can process dual-pipeline outputs without modification to its core evaluation logic.
3. WHEN the `SignalOutput` has a "dual_confirmed" flag, THE Signal_Engine SHALL set the recommendation confidence to the maximum of heuristic_confidence and probabilistic_P_up.
4. WHEN the `SignalOutput` has a "probabilistic_only" flag, THE Signal_Engine SHALL set the recommendation confidence to `probabilistic_P_up · 0.8` (20% confidence haircut for single-pipeline confirmation).
5. WHEN neither pipeline produces a BUY verdict, THE Signal_Engine SHALL not publish a trading recommendation to the queue (WATCH and SKIP verdicts are persisted for analysis but not forwarded to the trading engine).
---
### Requirement 13: Configuration and Feature Flags
**User Story:** As a platform operator, I want the dual-pipeline engine configurable via the existing `risk_configs` table and environment variables, so that I can tune thresholds, enable/disable individual pipelines, and adjust timeframe weights without code changes.
#### Acceptance Criteria
1. THE Signal_Engine SHALL support a `dual_pipeline_enabled` feature flag in `risk_configs` that toggles the entire dual-pipeline engine on or off, defaulting to false for safe rollout.
2. THE Signal_Engine SHALL support independent enable/disable flags for each pipeline: `heuristic_pipeline_enabled` and `probabilistic_pipeline_enabled`, both defaulting to true when the dual-pipeline engine is enabled.
3. THE Signal_Engine SHALL support configurable timeframe weights via a `timeframe_weights` JSON object in `risk_configs`, defaulting to `{"M30": 0.03, "H1": 0.07, "H4": 0.15, "D": 0.30, "W": 0.30, "M": 0.15}`.
4. THE Signal_Engine SHALL support configurable hard filter thresholds: `hard_filter_valuation_min` (default 0.3), `hard_filter_earnings_days` (default 5), and `hard_filter_macro_bias_skip` (default -1.0).
5. THE Signal_Engine SHALL support configurable verdict thresholds for both pipelines via `risk_configs` JSON, including heuristic confidence thresholds (BUY: 0.70, WATCH: 0.55) and probabilistic thresholds (P_up: 0.60, entropy: 0.90, EV_R: 1.5).
6. IF the `dual_pipeline_enabled` flag fails to read from the database, THEN THE Signal_Engine SHALL default to disabled (fail-safe behavior) and log a warning.
7. THE Signal_Engine SHALL log the active configuration at startup and on each configuration change for auditability.
---
### Requirement 14: Regime-Based Prior Engine
**User Story:** As a quantitative analyst, I want the probabilistic pipeline's prior probability to adapt based on the current market regime, so that the Bayesian inference starts from a regime-appropriate baseline rather than a fixed 0.50.
#### Acceptance Criteria
1. THE Probabilistic_Pipeline SHALL use the existing `classify_regime` function from `services/aggregation/regime.py` to determine the current market regime for each ticker.
2. THE Probabilistic_Pipeline SHALL map regime classifications to prior probabilities: trend_following with positive trend_indicator → 0.58 (bull), trend_following with negative trend_indicator → 0.42 (bear), mean_reversion → 0.50 (range), panic → 0.42 (bear), uncertainty → 0.50 (range).
3. THE Probabilistic_Pipeline SHALL convert the regime prior to log-odds before accumulating likelihood ratios: `logit(P_prior) = log(P_prior / (1 - P_prior))`.
4. WHEN market data is insufficient for regime classification (fewer than 100 days of price history), THE Probabilistic_Pipeline SHALL use the uncertainty prior of 0.50.
5. THE Probabilistic_Pipeline SHALL record the regime classification and prior probability in the `ProbabilisticResult` for auditability.
---
### Requirement 15: Database Schema for Signal Engine Output
**User Story:** As a platform operator, I want signal engine outputs persisted to a dedicated database table, so that historical evaluations are available for analysis, backtesting, and audit.
#### Acceptance Criteria
1. THE Signal_Engine SHALL persist each `SignalOutput` to a `signal_engine_outputs` table with columns for: id (UUID primary key), ticker (text), evaluated_at (timestamptz), price (numeric), heuristic_verdict (text), heuristic_confidence (numeric), heuristic_s_total (numeric), probabilistic_verdict (text), probabilistic_p_up (numeric), probabilistic_entropy (numeric), probabilistic_ev_r (numeric), delta_agreement (boolean), delta_confidence_delta (numeric), delta_reasons (JSONB), trade_plan (JSONB), full_output (JSONB), created_at (timestamptz).
2. THE Signal_Engine SHALL create an index on `(ticker, evaluated_at)` for efficient time-range queries per ticker.
3. THE Signal_Engine SHALL create an index on `evaluated_at` for efficient global time-range queries.
4. WHEN persisting fails due to a database error, THE Signal_Engine SHALL log the error and continue processing (persistence failure does not block signal emission to the trading queue).
---
### Requirement 16: Backward Compatibility and Migration Path
**User Story:** As a platform operator, I want the dual-pipeline engine to coexist with the existing single-pipeline aggregation, so that the rollout is incremental and reversible.
#### Acceptance Criteria
1. WHEN `dual_pipeline_enabled` is false, THE Signal_Engine SHALL not run, and the existing aggregation pipeline SHALL continue to operate unchanged.
2. WHEN `dual_pipeline_enabled` is true, THE Signal_Engine SHALL run alongside the existing aggregation pipeline, with the trading engine consuming `SignalOutput` from the dual-pipeline engine instead of `Recommendation` from the existing recommendation worker.
3. THE Signal_Engine SHALL reuse the existing `WeightedSignal`, `BayesianPosterior`, `RegimeClassification`, and `TrendSummary` data structures from `services/aggregation/` rather than duplicating them.
4. THE Signal_Engine SHALL reuse the existing `compute_signal_weight`, `compute_bayesian_posterior`, and `classify_regime` functions rather than reimplementing the underlying math.
5. THE Signal_Engine SHALL add the new `signal_engine_outputs` table via a new database migration without modifying existing tables.
6. THE Signal_Engine SHALL support running in "shadow mode" where both the existing pipeline and the dual-pipeline engine run, but only the existing pipeline's output is forwarded to the trading engine (dual-pipeline output is persisted for comparison only).
---
### Requirement 17: Property-Based Testing for Dual-Pipeline Correctness
**User Story:** As a developer, I want comprehensive property-based tests validating the mathematical correctness and structural invariants of the dual-pipeline engine, so that edge cases and numerical stability issues are caught before deployment.
#### Acceptance Criteria
1. THE test suite SHALL include property-based tests for the Fibonacci retracement formula verifying that `L(r) = SH - r·(SH - SL)` produces values in [SL, SH] for all r in [0, 1] and all SH > SL > 0.
2. THE test suite SHALL include property-based tests for the Bayesian log-odds update verifying that `logit(P_post) = logit(P_prior) + Σ log(LR_i)` round-trips correctly: converting P_prior to logit, adding log-LRs, and converting back via sigmoid produces a valid probability in (0, 1).
3. THE test suite SHALL include property-based tests for the entropy gate verifying that Shannon entropy is maximized at P_up = 0.5 and equals 0.0 at P_up = 0.0 or P_up = 1.0, and is symmetric around 0.5.
4. THE test suite SHALL include property-based tests for the signal correlation penalty verifying that the penalized posterior is always less than or equal to the unpenalized posterior for any signal set with correlated signals.
5. THE test suite SHALL include property-based tests for the multi-timeframe confluence score verifying monotonicity: activating a signal on an additional timeframe with non-zero weight always increases or maintains the confluence score.
6. THE test suite SHALL include property-based tests for the `SignalOutput` contract verifying round-trip serialization: `SignalOutput.model_validate_json(output.model_dump_json())` produces an equivalent object for all valid outputs.
7. THE test suite SHALL include property-based tests for the hard filter engine verifying that macro_bias = -1.0 always produces SKIP, valuation_score < 0.3 always produces SKIP, and earnings_proximity_days <= 5 always produces SKIP, regardless of all other input values.
8. THE test suite SHALL include property-based tests for the EV_R calculation verifying that `EV_R = P_up · E[win_R] - (1 - P_up) · 1.0` is monotonically increasing with P_up for fixed E[win_R] > 0.
@@ -0,0 +1,345 @@
# Implementation Plan: Dual-Pipeline Signal Engine
## Overview
Implement the dual-pipeline signal engine as a new service at `services/signal_engine/` that runs as an independent Kubernetes deployment. The engine evaluates both a heuristic (deterministic scoring) and probabilistic (Bayesian inference) pipeline concurrently per ticker per evaluation tick, producing independent BUY/WATCH/SKIP verdicts. Implementation proceeds incrementally: infrastructure first, then core models, signal library, pipelines, orchestration, integration, and deployment.
## Tasks
- [x] 1. Project scaffolding, configuration, and data models
- [x] 1.1 Create service directory structure and `__init__.py` files
- Create `services/signal_engine/` with all subdirectories per the design module structure
- Create `services/signal_engine/__init__.py`, `services/signal_engine/signals/__init__.py`
- _Requirements: 11.1, 13.1_
- [x] 1.2 Implement `models.py` — all Pydantic data models
- Define `OHLCVBar`, `NormalizedInput`, `OpenPositionState`, `SignalResult`, `SignalDirection`
- Define `ConfluenceSignal`, `Verdict`, `HeuristicResult`, `LikelihoodRatio`, `ProbabilisticResult`
- Define `DeltaResult`, `ExitSignal`, `ExitType`, `TradePlan`, `SignalOutput`
- All models must use Pydantic `BaseModel` with proper field constraints (`ge`, `le`)
- _Requirements: 1.1, 2.7, 5.7, 6.9, 9.5, 10.1, 10.5_
- [x] 1.3 Implement `config.py``SignalEngineConfig` and sub-configs
- Define `SignalEngineConfig` dataclass with all fields from the design
- Define `HardFilterConfig`, `HeuristicConfig`, `ProbabilisticConfig`, `ExitConfig` as derived sub-configs
- Implement `load_config()` that reads from `risk_configs` table + environment variables
- Default `dual_pipeline_enabled` to `False` (fail-safe)
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7_
- [x] 1.4 Add `QUEUE_SIGNAL_ENGINE` to `services/shared/redis_keys.py`
- Add `QUEUE_SIGNAL_ENGINE = "signal_engine"` constant
- _Requirements: 11.1_
- [x] 1.5 Write property test for `SignalOutput` round-trip serialization
- **Requirement 17.6: SignalOutput round-trip serialization**
- Generate arbitrary valid `SignalOutput` instances with Hypothesis
- Verify `SignalOutput.model_validate_json(output.model_dump_json())` produces equivalent object
- File: `tests/test_pbt_signal_engine_models.py`
- _Requirements: 10.5, 17.6_
- [x] 2. Input Normalizer and Hard Filter Engine
- [x] 2.1 Implement `normalizer.py` — Input Normalizer
- Implement `normalize_input(pool, ticker, config) -> NormalizedInput`
- Fetch OHLCV bars from `market_data_bars` for M30, H1, H4, D, W, M timeframes
- Fetch fundamental metrics (valuation_score, earnings_proximity_days) from company/trend data
- Fetch macro context (macro_bias) from `macro_impact_records` and `global_events`
- Fetch open position state from trading engine portfolio tables
- Populate sentinel values (`None`, empty list) for unavailable data with logged warnings
- Validate monotonically increasing timestamps within each timeframe series
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_
- [x] 2.2 Implement `hard_filter.py` — Hard Filter Engine
- Implement `evaluate_hard_filters(normalized, config) -> HardFilterResult`
- Check `macro_bias == -1.0` → SKIP with reason "macro_bias_negative"
- Check `valuation_score < 0.3` → SKIP with reason "valuation_below_threshold"
- Check `earnings_proximity_days <= 5` → SKIP with reason "earnings_block"
- Record all triggered filter reasons (not just first)
- Return `HardFilterResult` with `filtered: bool` and `reasons: list[str]`
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
- [x] 2.3 Write property tests for hard filter engine
- **Requirement 17.7: Hard filter determinism**
- Generate arbitrary `NormalizedInput` with `macro_bias = -1.0` → always SKIP
- Generate arbitrary `NormalizedInput` with `valuation_score < 0.3` → always SKIP
- Generate arbitrary `NormalizedInput` with `earnings_proximity_days <= 5` → always SKIP
- Verify these hold regardless of all other input values
- File: `tests/test_pbt_signal_engine_hard_filter.py`
- _Requirements: 4.1, 4.2, 4.3, 17.7_
- [x] 3. Checkpoint — Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
- [x] 4. Signal Library — Technical Signal Evaluators
- [x] 4.1 Implement `signals/base.py` — SignalEvaluator protocol
- Define `SignalEvaluator` protocol with `evaluate(bars, timeframe) -> SignalResult | None`
- Define common helper functions for swing high/low detection, lookback validation
- _Requirements: 2.6, 2.7_
- [x] 4.2 Implement `signals/fibonacci.py` — Fibonacci retracement evaluator
- Implement `L(r) = SH - r·(SH - SL)` for ratios [0.236, 0.382, 0.5, 0.618, 0.786]
- Detect swing high and swing low within the evaluation window
- Produce signal strength based on proximity of current price to retracement levels
- Return `None` with reason code when insufficient data
- _Requirements: 2.1, 2.6, 2.7_
- [x] 4.3 Write property test for Fibonacci retracement formula
- **Requirement 17.1: Fibonacci retracement bounds**
- For all `r` in [0, 1] and all `SH > SL > 0`, verify `L(r)` is in [SL, SH]
- File: `tests/test_pbt_signal_engine_fibonacci.py`
- _Requirements: 2.1, 17.1_
- [x] 4.4 Implement `signals/ma_stack.py` — Moving average stack evaluator
- Detect bullish alignment (MA_10 > MA_20 > MA_50 > MA_200)
- Detect bearish alignment (MA_10 < MA_20 < MA_50 < MA_200)
- Produce signal strength proportional to degree of alignment
- Return `None` when insufficient bars for MA_200 calculation
- _Requirements: 2.2, 2.6, 2.7_
- [x] 4.5 Implement `signals/rsi.py` — RSI evaluator
- Implement standard 14-period RSI formula
- Produce overbought signals (RSI > 70) and oversold signals (RSI < 30)
- Scale strength by distance from threshold
- Return `None` when fewer than 14 bars available
- _Requirements: 2.3, 2.6, 2.7_
- [x] 4.6 Implement `signals/cup_handle.py` — Cup & Handle pattern detector
- Identify cup formation (U-shaped price recovery) and handle (small consolidation)
- Produce signal with confidence proportional to pattern completeness
- Return `None` when insufficient data or no pattern detected
- _Requirements: 2.4, 2.6, 2.7_
- [x] 4.7 Implement `signals/elliott_wave.py` — Elliott Wave detector
- Identify impulse waves (5-wave structure) and corrective waves (3-wave structure)
- Produce signal with current wave position and projected direction
- Return `None` when insufficient data or ambiguous wave count
- _Requirements: 2.5, 2.6, 2.7_
- [x] 5. Multi-Timeframe Confluence Engine
- [x] 5.1 Implement `confluence.py` — Multi-Timeframe Engine
- Implement `compute_confluence(signal_results, weights) -> list[ConfluenceSignal]`
- Compute weighted confluence score: `C_confluence = Σ(w_tf · s_tf)`
- Apply minimum confluence threshold: discard signals triggering on < 2 timeframes
- Apply higher-timeframe anchor: discard signals without at least one of D, W, or M
- Return `ConfluenceSignal` objects with active timeframes and per-timeframe strengths
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_
- [x] 5.2 Write property test for confluence score monotonicity
- **Requirement 17.5: Confluence score monotonicity**
- Verify that activating a signal on an additional timeframe with non-zero weight always increases or maintains the confluence score
- File: `tests/test_pbt_signal_engine_confluence.py`
- _Requirements: 3.6, 17.5_
- [x] 6. Checkpoint — Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
- [x] 7. Heuristic Pipeline (Pipeline A)
- [x] 7.1 Implement `heuristic.py` — Heuristic Pipeline
- Implement `run_heuristic_pipeline(normalized, confluence_signals, config) -> HeuristicResult`
- Compute `S_total = S_company + S_macro + S_competitive` using existing `compute_signal_weight()`
- Compute confidence from source count, extraction confidence, signal agreement, contradiction penalty
- BUY verdict: confidence >= 0.70 AND S_total >= 1.2 AND valuation_score >= 0.5 AND macro_bias > 0 AND earnings_proximity_days > 5
- WATCH verdict: confidence >= 0.55 AND BUY conditions not fully met
- SKIP verdict: confidence < 0.55
- Emit `HeuristicResult` with all required fields and reasoning
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7_
- [x] 7.2 Write unit tests for heuristic pipeline verdict logic
- Test BUY threshold conditions
- Test WATCH threshold conditions
- Test SKIP conditions
- Test edge cases at threshold boundaries
- File: `tests/test_signal_engine_heuristic.py`
- _Requirements: 5.4, 5.5, 5.6_
- [x] 8. Probabilistic Pipeline (Pipeline B) and Correlation Penalty
- [x] 8.1 Implement `correlation.py` — Signal cluster classification and penalty
- Define `SignalCluster` enum: MOMENTUM, STRUCTURE, VOLATILITY, FUNDAMENTALS
- Implement `classify_signal(signal_type) -> SignalCluster`
- Implement `apply_correlation_penalty(likelihood_ratios) -> list[LikelihoodRatio]`
- Within-cluster decay: strongest LR at full weight, subsequent at 0.5^(n-1)
- No penalty across different clusters
- Single-signal clusters receive no penalty
- _Requirements: 7.1, 7.2, 7.3, 7.4_
- [x] 8.2 Implement `probabilistic.py` — Probabilistic Pipeline
- Implement `run_probabilistic_pipeline(normalized, confluence_signals, regime, config) -> ProbabilisticResult`
- Initialize regime-based prior: bull=0.58, range=0.50, bear=0.42
- Compute likelihood ratios: `P(sig|up) = h·s + (1-h)·(1-s)·0.5`, `LR = P(sig|up) / P(sig|down)`
- Apply correlation penalty via `apply_correlation_penalty()`
- Accumulate via log-odds: `logit(P_post) = logit(P_prior) + Σ log(LR_i)`
- Compute Shannon entropy and apply entropy gating (H > 0.95 → SKIP)
- Compute `EV_R = P_up · E[win_R] - (1 - P_up) · 1.0`
- BUY: P_up >= 0.60 AND entropy <= 0.90 AND EV_R >= 1.5 AND macro_bias > 0 AND valuation_score >= 0.5
- WATCH: P_up >= 0.55 AND entropy <= 0.95 AND BUY conditions not fully met
- SKIP: all other cases
- Use existing `classify_regime()` from `services/aggregation/regime.py`
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9, 14.1, 14.2, 14.3, 14.4, 14.5_
- [x] 8.3 Write property test for Bayesian log-odds round-trip
- **Requirement 17.2: Bayesian log-odds update correctness**
- Verify `logit(P_post) = logit(P_prior) + Σ log(LR_i)` round-trips correctly
- Converting P_prior to logit, adding log-LRs, converting back via sigmoid produces valid probability in (0, 1)
- File: `tests/test_pbt_signal_engine_bayesian.py`
- _Requirements: 6.3, 17.2_
- [x] 8.4 Write property test for entropy gate
- **Requirement 17.3: Entropy gate properties**
- Verify Shannon entropy is maximized at P_up = 0.5
- Verify entropy equals 0.0 at P_up = 0.0 or P_up = 1.0
- Verify entropy is symmetric around 0.5
- File: `tests/test_pbt_signal_engine_bayesian.py`
- _Requirements: 6.4, 17.3_
- [x] 8.5 Write property test for signal correlation penalty
- **Requirement 17.4: Correlation penalty reduces confidence**
- Verify penalized posterior is always <= unpenalized posterior for any signal set with correlated signals
- File: `tests/test_pbt_signal_engine_correlation.py`
- _Requirements: 7.5, 17.4_
- [x] 8.6 Write property test for EV_R monotonicity
- **Requirement 17.8: EV_R monotonically increasing with P_up**
- Verify `EV_R = P_up · E[win_R] - (1 - P_up) · 1.0` is monotonically increasing with P_up for fixed E[win_R] > 0
- File: `tests/test_pbt_signal_engine_bayesian.py`
- _Requirements: 6.5, 17.8_
- [x] 9. Checkpoint — Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
- [x] 10. Exit Engine
- [x] 10.1 Implement `exit_engine.py` — Exit Engine
- Implement `evaluate_exits(positions, current_prices, config) -> list[ExitSignal]`
- Check stop_loss hit → EXIT_FULL with reason "stop_hit"
- Check target_1 hit → EXIT_HALF with reason "target_1_hit"
- Check target_2 hit → EXIT_FULL with reason "target_2_hit"
- Trailing stop: activate after EXIT_HALF at `current_price - ATR · trailing_multiplier`
- Trailing stop ratchets upward only (never moves down)
- Trailing stop hit → EXIT_FULL with reason "trailing_stop_hit"
- _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7_
- [x] 10.2 Write unit tests for exit engine
- Test stop_loss trigger
- Test target_1 partial exit
- Test target_2 full exit
- Test trailing stop activation and ratchet behavior
- File: `tests/test_signal_engine_exit.py`
- _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5_
- [x] 11. Delta Analyzer and Output Formatter
- [x] 11.1 Implement `delta.py` — Delta Analyzer
- Implement `analyze_delta(heuristic, probabilistic, redis, ticker) -> DeltaResult`
- Compute agreement flag (both verdicts identical)
- Compute confidence delta: `|heuristic_confidence - probabilistic_P_up|`
- Record disagreement reasons when verdicts differ
- Track rolling 100-evaluation agreement rate in Redis
- Log warning when agreement rate drops below 0.50
- _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6_
- [x] 11.2 Implement `formatter.py` — Output Formatter
- Implement `format_output(ticker, price, heuristic, probabilistic, delta, exit_signals, config) -> SignalOutput`
- Both BUY → `dual_confirmed`, full position sizing
- Probabilistic-only BUY → `probabilistic_only`, 50% position sizing
- Heuristic-only BUY → standard position sizing
- No BUY → no trade_plan (WATCH/SKIP persisted for analysis)
- Implement `signal_output_to_recommendation(output) -> Recommendation`
- Map `SignalOutput` to existing `Recommendation` schema for trading engine compatibility
- Dual confirmed: confidence = max(heuristic_confidence, probabilistic_P_up)
- Probabilistic only: confidence = probabilistic_P_up · 0.8 (20% haircut)
- _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 12.1, 12.2, 12.3, 12.4, 12.5_
- [x] 11.3 Write unit tests for output formatter
- Test dual_confirmed trade plan generation
- Test probabilistic_only trade plan with 50% sizing
- Test heuristic-only trade plan
- Test no-BUY case (no trade_plan)
- Test `signal_output_to_recommendation` mapping
- File: `tests/test_signal_engine_formatter.py`
- _Requirements: 10.2, 10.3, 10.4, 12.3, 12.4_
- [x] 12. Orchestrator, Persistence, and Main Entry Point
- [x] 12.1 Implement `persistence.py` — Database persistence
- Implement `persist_signal_output(pool, output) -> None`
- Insert into `signal_engine_outputs` table
- Log and continue on database errors (non-blocking)
- _Requirements: 15.1, 15.4_
- [x] 12.2 Implement `worker.py` — Top-level orchestrator
- Implement `evaluate_tick(pool, redis, ticker, config) -> SignalOutput | None`
- Step 1: Normalize inputs (single fetch, shared reference)
- Step 2: Evaluate exit conditions for open positions
- Step 3: Run hard filters (short-circuit if filtered)
- Step 4: Evaluate signals across timeframes via Signal Library
- Step 5: Compute confluence
- Step 6: Classify regime via existing `classify_regime()`
- Step 7: Run both pipelines concurrently via `asyncio.gather` with exception handling
- Step 8: Compute delta analysis
- Step 9: Format output
- Step 10: Persist to database and publish to Redis queue
- Catch pipeline exceptions → SKIP verdict for failed pipeline, other continues
- Measure and log wall-clock execution time per pipeline
- _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6_
- [x] 12.3 Implement `main.py` — Entry point with asyncio event loop
- Connect to PostgreSQL (asyncpg pool) and Redis (redis.asyncio)
- Load config from `risk_configs` table
- Log active configuration at startup
- Poll `stonks:queue:signal_engine` queue indefinitely
- Check `dual_pipeline_enabled` flag; if disabled, sleep and retry
- On config read failure, default to disabled (fail-safe)
- Support shadow mode (persist but don't forward to trading queue)
- _Requirements: 13.1, 13.6, 13.7, 16.1, 16.6_
- [x] 12.4 Write integration tests for worker orchestration
- Test full tick evaluation with mocked DB/Redis
- Test pipeline failure isolation (one fails, other completes)
- Test hard filter short-circuit
- Test shadow mode behavior
- File: `tests/test_signal_engine_worker.py`
- _Requirements: 11.3, 16.6_
- [x] 13. Checkpoint — Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
- [x] 14. Database migration and infrastructure
- [x] 14.1 Create database migration `infra/migrations/039_signal_engine_outputs.sql`
- Create `signal_engine_outputs` table per the design schema
- Create index on `(ticker, evaluated_at)` for per-ticker time-range queries
- Create index on `evaluated_at` for global time-range queries
- Create index on `(heuristic_verdict, probabilistic_verdict)` for verdict filtering
- _Requirements: 15.1, 15.2, 15.3_
- [x] 14.2 Add signal engine service to Helm chart
- Add `signalEngine` entry to `infra/helm/stonks-oracle/values.yaml`
- Configure: replicas=1, command=`python -m services.signal_engine.main`, tier=processing
- Set resource requests/limits per design (100m/128Mi → 500m/256Mi)
- Reference existing secrets: `stonks-core-secrets`, `stonks-market-secrets`
- _Requirements: 11.1, 13.1_
- [x] 15. Trading engine integration and backward compatibility
- [x] 15.1 Wire signal engine output to trading engine queue
- Publish `SignalOutput` (mapped to `Recommendation`) to `stonks:queue:trading_decisions`
- Only publish when at least one pipeline produces BUY verdict
- WATCH/SKIP verdicts persisted for analysis but not forwarded
- Ensure trading engine can consume without modification via `signal_output_to_recommendation()`
- _Requirements: 12.1, 12.2, 12.5, 16.2_
- [x] 15.2 Ensure backward compatibility with existing pipeline
- Verify `dual_pipeline_enabled=false` means signal engine does not run
- Verify existing aggregation pipeline operates unchanged when flag is off
- Reuse existing `WeightedSignal`, `BayesianPosterior`, `RegimeClassification` (import, don't duplicate)
- Reuse existing `compute_signal_weight`, `compute_bayesian_posterior`, `classify_regime` functions
- No modifications to existing tables (new migration only adds new table)
- _Requirements: 16.1, 16.2, 16.3, 16.4, 16.5_
- [x] 16. Final checkpoint — Ensure all tests pass
- Ensure all tests pass, ask the user if questions arise.
## Notes
- Tasks marked with `*` are optional and can be skipped for faster MVP
- Each task references specific requirements for traceability
- Checkpoints ensure incremental validation between major phases
- Property-based tests use Hypothesis with `@settings(max_examples=100)` per project conventions
- PBT test files are prefixed `test_pbt_*` per project conventions
- The service reuses existing math functions from `services/aggregation/` — no reimplementation
- All configuration is loaded from `risk_configs` table with fail-safe defaults
- Shadow mode allows running alongside existing pipeline without affecting trading decisions