feat: math core v3 engine upgrade
This commit is contained in:
Celes Renata
@@ -0,0 +1 @@
|
||||
{"specId": "ff0d03d7-3469-4551-bf05-15295b735c83", "workflowType": "requirements-first", "specType": "feature"}
|
||||
@@ -0,0 +1,851 @@
|
||||
# Design Document: Math Core v3 Engine
|
||||
|
||||
## Overview
|
||||
|
||||
The v3 Calibrated Evidence Engine replaces the current dual-mode pipeline (heuristic + probabilistic) with a principled Bayesian evidence accumulation system. The upgrade transforms the signal processing core from weighted-sentiment averaging to:
|
||||
|
||||
```
|
||||
EvidenceUnit → calibrated reliability (q_i) → log-likelihood ratio (LLR_i)
|
||||
→ correlation-adjusted cluster LLR → posterior P_up → return distribution → EV/risk decision
|
||||
```
|
||||
|
||||
**Key design goals:**
|
||||
- Replace arbitrary weight products with calibrated probabilistic evidence
|
||||
- Prevent correlated articles from inflating evidence counts
|
||||
- Make confidence multiplicative so one bad dimension suppresses the trade
|
||||
- Size positions using fractional Kelly criterion under hard risk caps
|
||||
- Preserve the existing three-layer architecture and service boundaries
|
||||
- Gate behind `v3_engine_enabled` feature flag with heuristic fallback
|
||||
|
||||
**What changes vs. what stays:**
|
||||
- The `WeightedSignal` abstraction remains as an intermediate before LLR conversion
|
||||
- All output goes into existing JSONB metadata columns (no new migrations)
|
||||
- Service file boundaries are preserved; internals are upgraded
|
||||
- The heuristic pipeline stays as fallback, controlled by a single flag read per aggregation cycle
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Input Layer
|
||||
CS[Company Signals]
|
||||
MS[Macro Signals]
|
||||
XS[Competitive Signals]
|
||||
end
|
||||
|
||||
subgraph Normalization
|
||||
EU[EvidenceUnit Normalization]
|
||||
end
|
||||
|
||||
subgraph Scoring Pipeline
|
||||
QI[Calibrated Reliability q_i]
|
||||
LLR[LLR Conversion]
|
||||
end
|
||||
|
||||
subgraph Clustering
|
||||
CL[Correlation-Aware Clustering]
|
||||
NEFF[n_eff Computation]
|
||||
CLLR[Cluster LLR]
|
||||
end
|
||||
|
||||
subgraph Posterior Assembly
|
||||
REG[Regime Detection v3]
|
||||
POST[Log-Odds Posterior P_up]
|
||||
CON[LLR Entropy Contradiction]
|
||||
CONF[Multiplicative Confidence]
|
||||
end
|
||||
|
||||
subgraph Decision Layer
|
||||
PROJ[Posterior State Projection]
|
||||
RET[Return Distribution / EV Gate]
|
||||
ELIG[Regime-Aware Eligibility]
|
||||
KELLY[Fractional Kelly Sizing]
|
||||
STOP[Regime-Aware Stops]
|
||||
HEAT[Stop-Defined Portfolio Heat]
|
||||
end
|
||||
|
||||
subgraph Quality & Control
|
||||
DQ[Data Quality v3]
|
||||
FF[Feature Flag Router]
|
||||
TIER[Risk Tier Auto-Adjustment]
|
||||
end
|
||||
|
||||
CS --> EU
|
||||
MS --> EU
|
||||
XS --> EU
|
||||
EU --> QI
|
||||
QI --> LLR
|
||||
LLR --> CL
|
||||
CL --> NEFF
|
||||
NEFF --> CLLR
|
||||
REG --> POST
|
||||
CLLR --> POST
|
||||
POST --> CON
|
||||
POST --> CONF
|
||||
CONF --> PROJ
|
||||
PROJ --> RET
|
||||
RET --> ELIG
|
||||
ELIG --> KELLY
|
||||
KELLY --> STOP
|
||||
KELLY --> HEAT
|
||||
DQ --> CONF
|
||||
FF --> EU
|
||||
TIER --> KELLY
|
||||
```
|
||||
|
||||
### Service File Mapping
|
||||
|
||||
| Service File | v3 Responsibility |
|
||||
|---|---|
|
||||
| `services/aggregation/scoring.py` | EvidenceUnit normalization, q_i pipeline, LLR conversion |
|
||||
| `services/aggregation/bayesian.py` | Posterior assembly via log-odds, P_up, strength |
|
||||
| `services/aggregation/contradiction.py` | LLR entropy contradiction score |
|
||||
| `services/aggregation/regime.py` | Regime detection v3 (ATR-normalized trend_z) |
|
||||
| `services/aggregation/interpolation.py` | Noisy-OR macro exposure, LLR emission |
|
||||
| `services/aggregation/signal_propagation.py` | Correlation-shrunk competitive propagation |
|
||||
| `services/aggregation/projection.py` | Posterior state A_t, regime-aware decay |
|
||||
| `services/aggregation/worker.py` | Orchestration, clustering, n_eff, confidence assembly |
|
||||
| `services/recommendation/eligibility.py` | EV gate, regime-aware eligibility, mode escalation |
|
||||
| `services/trading/position_sizer.py` | Fractional Kelly sizing under caps |
|
||||
| `services/trading/stop_loss_manager.py` | Regime-aware stop/TP, trailing activation |
|
||||
| `services/risk/engine.py` | Stop-defined heat, tier auto-adjustment |
|
||||
|
||||
### Feature Flag Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Worker as aggregation/worker.py
|
||||
participant DB as risk_configs table
|
||||
participant V3 as v3 Pipeline
|
||||
participant Heuristic as Heuristic Pipeline
|
||||
|
||||
Worker->>DB: SELECT v3_engine_enabled
|
||||
alt v3_engine_enabled = True
|
||||
Worker->>V3: Run v3 pipeline
|
||||
V3-->>Worker: Posterior + confidence + EV
|
||||
alt Unhandled error
|
||||
V3-->>Worker: Exception
|
||||
Worker->>Heuristic: Fallback to heuristic
|
||||
Worker->>Worker: Log error, record fallback in metadata
|
||||
end
|
||||
else v3_engine_enabled = False or DB error
|
||||
Worker->>Heuristic: Run heuristic pipeline
|
||||
end
|
||||
```
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### 1. EvidenceUnit Dataclass (`scoring.py`)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class EvidenceUnit:
|
||||
"""Canonical normalized signal representation for v3 pipeline."""
|
||||
symbol: str
|
||||
layer: str # "company" | "macro" | "competitive"
|
||||
event_type: str
|
||||
source_id: str
|
||||
source_group: str
|
||||
timestamp: datetime
|
||||
horizon: str # "intraday" | "1d" | "7d" | "30d" | "90d"
|
||||
direction: int # -1, 0, +1
|
||||
sentiment_strength: float # [0, 1]
|
||||
impact: float # [0, 1]
|
||||
extraction_conf: float # [0, 1]
|
||||
source_cred: float # [0, 1]
|
||||
novelty: float # [0, 1]
|
||||
event_base_rate: float # (0, 1]
|
||||
cluster_id: str
|
||||
```
|
||||
|
||||
**Normalization functions** — one per layer:
|
||||
- `normalize_company_signal(impact_row, ...) -> EvidenceUnit`
|
||||
- `normalize_macro_signal(macro_impact_record, global_event, ...) -> EvidenceUnit`
|
||||
- `normalize_competitive_signal(competitive_signal_record, ...) -> EvidenceUnit`
|
||||
|
||||
Each validates required fields (symbol, timestamp, source_id), substitutes 0.5 for missing optional numeric fields, and assigns direction from sentiment/impact_direction strings.
|
||||
|
||||
### 2. Calibrated Reliability Pipeline (`scoring.py`)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ReliabilityComponents:
|
||||
q_ext: float
|
||||
q_source: float
|
||||
q_recency: float
|
||||
q_uniqueness: float
|
||||
q_i: float # final combined reliability
|
||||
|
||||
def compute_v3_reliability(
|
||||
unit: EvidenceUnit,
|
||||
source_stats: SourceStats,
|
||||
cluster_position: int, # duplicate_count_before
|
||||
reference_time: datetime,
|
||||
) -> ReliabilityComponents: ...
|
||||
```
|
||||
|
||||
Sub-computations:
|
||||
- `q_ext = sigmoid(8.0 * (extraction_conf - 0.55))`
|
||||
- `q_source = clamp((E[theta_s] - 0.50) / 0.35, 0, 1)` with Beta(alpha_0+hits, beta_0+misses)
|
||||
- `q_recency = 2^(-age_hours / tau_adaptive)` with adaptive half-life
|
||||
- `q_uniqueness = clamp(0.5 + 0.5 * novelty, 0.5, 1.0) * (1 / sqrt(1 + dup_count))`
|
||||
- `q_i = clamp(q_ext * q_source * source_cred * q_recency * q_uniqueness, 0, 1)`
|
||||
|
||||
### 3. LLR Conversion (`scoring.py`)
|
||||
|
||||
```python
|
||||
def compute_llr(unit: EvidenceUnit, q_i: float) -> float:
|
||||
"""Convert calibrated reliability to log-likelihood ratio."""
|
||||
p_correct = clamp(0.50 + 0.35 * q_i * unit.impact * unit.sentiment_strength, 0.501, 0.85)
|
||||
if unit.direction == 0:
|
||||
return 0.0
|
||||
return unit.direction * math.log(p_correct / (1 - p_correct))
|
||||
```
|
||||
|
||||
### 4. Correlation-Aware Clustering (`worker.py`)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class EvidenceCluster:
|
||||
cluster_id: str
|
||||
units: list[EvidenceUnit]
|
||||
llrs: list[float]
|
||||
n_eff: float
|
||||
cluster_llr: float
|
||||
|
||||
def cluster_evidence(units: list[EvidenceUnit], llrs: list[float]) -> list[EvidenceCluster]:
|
||||
"""Group by (symbol, horizon, event_type, source_group, time_bucket)."""
|
||||
...
|
||||
|
||||
def compute_n_eff(llrs: list[float], correlations: list[list[float]]) -> float:
|
||||
"""n_eff = (sum w_i)^2 / (sum w_i^2 + 2*sum_{i<j} rho_ij*w_i*w_j)"""
|
||||
...
|
||||
|
||||
def compute_cluster_llr(llrs: list[float], n_eff: float) -> float:
|
||||
"""LLR_c = clamp(weighted_mean(LLR_i, |LLR_i|) * sqrt(n_eff), -2.5, 2.5)"""
|
||||
...
|
||||
```
|
||||
|
||||
Default pairwise correlations:
|
||||
| Relationship | rho |
|
||||
|---|---:|
|
||||
| Same wire/story/source group | 0.80 |
|
||||
| Same event, different publisher | 0.50 |
|
||||
| Same theme, different event | 0.25 |
|
||||
| Independent events | 0.00 |
|
||||
|
||||
### 5. Posterior Assembly (`bayesian.py`)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class V3Posterior:
|
||||
p_up: float # sigmoid(log_odds)
|
||||
p_down: float # 1 - p_up
|
||||
log_odds: float # logit(P_prior) + sum(gamma * LLR_c)
|
||||
strength: float # abs(2 * p_up - 1)
|
||||
direction: str # bullish | bearish | neutral
|
||||
n_eff_total: float
|
||||
regime: str
|
||||
|
||||
def compute_v3_posterior(
|
||||
clusters: list[EvidenceCluster],
|
||||
regime: RegimeClassification,
|
||||
p_prior: float = 0.50,
|
||||
) -> V3Posterior: ...
|
||||
```
|
||||
|
||||
Direction thresholds by regime:
|
||||
| Regime | Bullish if P_up >= | Bearish if P_up <= |
|
||||
|---|---:|---:|
|
||||
| panic | 0.68 | 0.32 |
|
||||
| trend_following | 0.60 | 0.40 |
|
||||
| mean_reversion | 0.63 | 0.37 |
|
||||
| uncertainty | 0.65 | 0.35 |
|
||||
|
||||
### 6. Regime Detection v3 (`regime.py`)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class V3RegimeClassification:
|
||||
regime: MarketRegime
|
||||
trend_z: float # (EMA_20 - EMA_100) / ATR_20
|
||||
vol_ratio: float # sigma_20 / sigma_100
|
||||
evidence_multiplier: float # gamma_regime
|
||||
confidence_multiplier: float
|
||||
phi_decay: float # for projection
|
||||
atr_multiplier: float # for stops
|
||||
|
||||
def classify_regime_v3(
|
||||
closing_prices: list[float],
|
||||
daily_returns: list[float],
|
||||
atr_20: float,
|
||||
) -> V3RegimeClassification: ...
|
||||
```
|
||||
|
||||
Classification rules (in priority order):
|
||||
1. **Panic**: vol_ratio > 1.5 OR abs(trend_z) > 2.5
|
||||
2. **Trend following**: abs(trend_z) >= 0.75 AND vol_ratio < 1.3
|
||||
3. **Mean reversion**: abs(trend_z) < 0.50 AND vol_ratio < 1.0
|
||||
4. **Uncertainty**: all other cases
|
||||
|
||||
### 7. LLR Entropy Contradiction (`contradiction.py`)
|
||||
|
||||
```python
|
||||
def compute_v3_contradiction(clusters: list[EvidenceCluster]) -> float:
|
||||
"""
|
||||
E_pos = sum(max(LLR_c, 0))
|
||||
E_neg = sum(max(-LLR_c, 0))
|
||||
H = -f_pos*log2(f_pos) - f_neg*log2(f_neg)
|
||||
volume_factor = 1 - exp(-E_total / 3.0)
|
||||
return H * volume_factor
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### 8. Multiplicative Confidence (`worker.py`)
|
||||
|
||||
```python
|
||||
def compute_v3_confidence(
|
||||
n_eff_total: float,
|
||||
q_values: list[float],
|
||||
llrs: list[float],
|
||||
strength: float,
|
||||
regime_confidence_mult: float,
|
||||
contradiction: float,
|
||||
data_quality: float,
|
||||
) -> float:
|
||||
"""
|
||||
C_evidence = 1 - exp(-n_eff_total / 5.0)
|
||||
C_quality = weighted_mean(q_i, |LLR_i|)
|
||||
confidence = clamp(
|
||||
C_evidence * sqrt(C_quality) * sqrt(max(strength, 0.05))
|
||||
* regime_confidence_mult * (1 - contradiction) * data_quality,
|
||||
0, 1
|
||||
)
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### 9. Noisy-OR Macro Exposure (`interpolation.py`)
|
||||
|
||||
```python
|
||||
def compute_normalized_macro_exposure(overlaps: dict[str, float]) -> float:
|
||||
"""
|
||||
E_raw = 1 - prod(1 - w_k * O_k)
|
||||
E_max = 1 - prod(1 - w_k)
|
||||
return E_raw / E_max
|
||||
"""
|
||||
...
|
||||
|
||||
def compute_macro_llr(
|
||||
macro_impact: float,
|
||||
event_confidence: float,
|
||||
q_recency: float,
|
||||
macro_direction: int,
|
||||
) -> float:
|
||||
"""
|
||||
p_macro = clamp(0.50 + 0.30 * macro_impact * event_confidence * q_recency, 0.501, 0.80)
|
||||
return macro_direction * log(p_macro / (1 - p_macro))
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### 10. Correlation-Shrunk Competitive Propagation (`signal_propagation.py`)
|
||||
|
||||
```python
|
||||
def compute_shrunk_correlation(
|
||||
rho_rolling: float,
|
||||
n_observations: int,
|
||||
same_sector: bool,
|
||||
) -> float:
|
||||
"""
|
||||
rho_prior = 0.30 if same_sector else 0.10
|
||||
rho_shrunk = (n/(n+30)) * rho_rolling + (30/(n+30)) * rho_prior
|
||||
return max(rho_shrunk, 0)
|
||||
"""
|
||||
...
|
||||
|
||||
def compute_competitive_llr(
|
||||
llr_source: float,
|
||||
rho_effective: float,
|
||||
d_network: int,
|
||||
pattern_confidence: float,
|
||||
) -> float:
|
||||
"""
|
||||
attenuation = rho_effective * exp(-0.85 * d_network)
|
||||
return clamp(llr_source * attenuation * pattern_confidence, -1.25, 1.25)
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### 11. Posterior State Projection (`projection.py`)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class V3ProjectionState:
|
||||
a_t: float # accumulated evidence state
|
||||
p_up_projected: float # sigmoid(logit(P_prior) + phi^h * A_t)
|
||||
projected_strength: float
|
||||
diverges: bool
|
||||
phi_regime: float
|
||||
|
||||
def compute_v3_projection(
|
||||
a_prev: float,
|
||||
cluster_llrs: list[float],
|
||||
regime: V3RegimeClassification,
|
||||
p_prior: float,
|
||||
projection_horizon: int,
|
||||
known_catalyst_llr: float = 0.0,
|
||||
) -> V3ProjectionState: ...
|
||||
```
|
||||
|
||||
Regime decay factors (phi):
|
||||
| Regime | phi |
|
||||
|---|---:|
|
||||
| panic | 0.35 |
|
||||
| trend_following | 0.80 |
|
||||
| mean_reversion | 0.55 |
|
||||
| uncertainty | 0.50 |
|
||||
|
||||
### 12. Return Distribution and EV Gate (`eligibility.py`)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ReturnDistribution:
|
||||
sigma_h: float # realized_vol_20d * sqrt(horizon_days / 252)
|
||||
mu_h: float # tanh(A_projected / 3.0) * confidence * sigma_h
|
||||
ev_long: float # mu_h - costs - 0.10 * CVaR_5
|
||||
min_edge: float # regime-specific minimum edge
|
||||
eligible: bool
|
||||
|
||||
def compute_return_distribution(
|
||||
a_projected: float,
|
||||
confidence: float,
|
||||
realized_vol_20d: float,
|
||||
horizon_days: int,
|
||||
costs: float,
|
||||
regime: str,
|
||||
) -> ReturnDistribution: ...
|
||||
```
|
||||
|
||||
### 13. Fractional Kelly Position Sizing (`position_sizer.py`)
|
||||
|
||||
```python
|
||||
def compute_kelly_sizing(
|
||||
p_win: float, # P_up from posterior
|
||||
b: float, # reward ratio: clamp(1.2 + 2*conf + str - contra, 1.2, 3.0)
|
||||
confidence: float,
|
||||
data_quality: float,
|
||||
contradiction: float,
|
||||
max_position_pct: float,
|
||||
available_caps: dict[str, float], # sector, correlation, heat capacities
|
||||
) -> float:
|
||||
"""
|
||||
f_kelly = (p_win * b - (1 - p_win)) / b
|
||||
portfolio_pct = clamp(max(0, f_kelly) * 0.25 * confidence * data_quality * (1 - contradiction), 0, max_position_pct)
|
||||
Apply min of all capacity constraints.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### 14. Stop-Defined Portfolio Heat (`risk/engine.py`)
|
||||
|
||||
```python
|
||||
def compute_portfolio_heat(
|
||||
positions: list[OpenPosition],
|
||||
stop_distances: dict[str, float],
|
||||
) -> float:
|
||||
"""risk_dollars = position_value * stop_distance_pct; heat = sum(risk_dollars)"""
|
||||
...
|
||||
|
||||
def check_heat_capacity(
|
||||
current_heat: float,
|
||||
new_risk_dollars: float,
|
||||
max_heat_pct: float,
|
||||
portfolio_value: float,
|
||||
) -> bool: ...
|
||||
```
|
||||
|
||||
### 15. Data Quality v3 (`worker.py`)
|
||||
|
||||
```python
|
||||
def compute_v3_data_quality(
|
||||
units: list[EvidenceUnit],
|
||||
extraction_failure_rate: float,
|
||||
age_newest_hours: float,
|
||||
n_source_types: int,
|
||||
) -> float:
|
||||
"""
|
||||
Q_parse = 1 - extraction_failure_rate
|
||||
Q_conf = weighted_mean(extraction_conf, impact)
|
||||
Q_fresh = exp(-age_newest_hours / 168)
|
||||
Q_coverage = 1 - exp(-N_valid / 5)
|
||||
Q_diversity = min(1, log2(1 + N_source_types) / log2(4))
|
||||
return clamp(Q_parse * sqrt(Q_conf) * Q_fresh * Q_coverage * Q_diversity, 0, 1)
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
### 16. Risk Tier Auto-Adjustment (`risk/engine.py`)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TierMetrics:
|
||||
profit_factor_30d: float
|
||||
max_drawdown_30d: float
|
||||
calibration_error: float
|
||||
realized_sharpe_30d: float
|
||||
n_trades_30d: int
|
||||
reserve_pool_pct: float
|
||||
|
||||
def evaluate_tier_adjustment(metrics: TierMetrics) -> str:
|
||||
"""Returns 'upgrade' | 'downgrade' | 'hold'"""
|
||||
...
|
||||
```
|
||||
|
||||
## Data Models
|
||||
|
||||
### EvidenceUnit (frozen dataclass)
|
||||
|
||||
| Field | Type | Range | Source |
|
||||
|---|---|---|---|
|
||||
| symbol | str | — | Required from signal |
|
||||
| layer | str | company/macro/competitive | Set during normalization |
|
||||
| event_type | str | — | catalyst_type or impact_type |
|
||||
| source_id | str | — | document_id or event_id |
|
||||
| source_group | str | — | publisher / "macro" / "competitive" |
|
||||
| timestamp | datetime | — | published_at |
|
||||
| horizon | str | intraday/1d/7d/30d/90d | window or estimated_duration mapping |
|
||||
| direction | int | -1, 0, +1 | sentiment/direction mapping |
|
||||
| sentiment_strength | float | [0, 1] | impact_score or sentiment confidence |
|
||||
| impact | float | [0, 1] | impact_score |
|
||||
| extraction_conf | float | [0, 1] | confidence field |
|
||||
| source_cred | float | [0, 1] | source_credibility |
|
||||
| novelty | float | [0, 1] | novelty_score |
|
||||
| event_base_rate | float | (0, 1] | lookup by event_type |
|
||||
| cluster_id | str | — | computed hash of clustering key |
|
||||
|
||||
### V3 Posterior Output (stored in JSONB metadata)
|
||||
|
||||
```json
|
||||
{
|
||||
"v3_posterior": {
|
||||
"p_up": 0.64,
|
||||
"p_down": 0.36,
|
||||
"log_odds": 0.58,
|
||||
"strength": 0.28,
|
||||
"confidence": 0.61,
|
||||
"contradiction": 0.18,
|
||||
"n_eff": 5.7,
|
||||
"data_quality": 0.82,
|
||||
"regime": "trend_following"
|
||||
},
|
||||
"v3_return_model": {
|
||||
"mu_h": 0.012,
|
||||
"sigma_h": 0.041,
|
||||
"ev_long": 0.007,
|
||||
"min_edge": 0.0035
|
||||
},
|
||||
"v3_explainability": {
|
||||
"top_positive_clusters": [...],
|
||||
"top_negative_clusters": [...],
|
||||
"suppression_reasons": [],
|
||||
"risk_adjustments": []
|
||||
},
|
||||
"pipeline_mode": "v3"
|
||||
}
|
||||
```
|
||||
|
||||
### Source Statistics (for q_source computation)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class SourceStats:
|
||||
source_id: str
|
||||
hits: int = 0 # correct directional predictions
|
||||
misses: int = 0 # incorrect directional predictions
|
||||
alpha_0: int = 3 # prior
|
||||
beta_0: int = 3 # prior
|
||||
```
|
||||
|
||||
### Regime Parameters Table
|
||||
|
||||
| Regime | gamma (evidence) | confidence_mult | phi (decay) | ATR_mult (stops) | min_edge |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| panic | 0.70 | 0.70 | 0.35 | 2.5 | 0.0100 |
|
||||
| trend_following | 1.10 | 1.00 | 0.80 | 1.8 | 0.0035 |
|
||||
| mean_reversion | 0.90 | 0.95 | 0.55 | 1.4 | 0.0050 |
|
||||
| uncertainty | 0.80 | 0.85 | 0.50 | 2.0 | 0.0075 |
|
||||
|
||||
### Eligibility Thresholds (Regime-Specific)
|
||||
|
||||
| Regime | confidence_min | contradiction_max | strength_min |
|
||||
|---|---:|---:|---:|
|
||||
| panic | 0.70 | 0.25 | 0.36 |
|
||||
| trend_following | 0.55 | 0.40 | 0.20 |
|
||||
| mean_reversion | 0.60 | 0.35 | 0.26 |
|
||||
| uncertainty | 0.65 | 0.30 | 0.30 |
|
||||
|
||||
|
||||
## Correctness Properties
|
||||
|
||||
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
|
||||
|
||||
### Property 1: Reliability q_i is bounded in [0, 1]
|
||||
|
||||
*For any* valid EvidenceUnit with extraction_conf in [0,1], source_cred in [0,1], novelty in [0,1], any non-negative age_hours, and any non-negative duplicate_count_before, the computed q_i SHALL be in the range [0.0, 1.0].
|
||||
|
||||
**Validates: Requirements 2.8, 21.1**
|
||||
|
||||
### Property 2: p_correct is bounded in [0.501, 0.85]
|
||||
|
||||
*For any* valid q_i in [0, 1], impact in [0, 1], and sentiment_strength in [0, 1], the computed p_correct SHALL be in the range [0.501, 0.85].
|
||||
|
||||
**Validates: Requirements 3.1, 21.2**
|
||||
|
||||
### Property 3: LLR sign matches direction and magnitude is bounded
|
||||
|
||||
*For any* valid signal with direction in {-1, +1}, the computed LLR SHALL have the same sign as direction, with absolute magnitude in [ln(0.501/0.499), ln(0.85/0.15)] ≈ [0.004, 1.735].
|
||||
|
||||
**Validates: Requirements 3.2, 3.4, 3.5, 3.6, 21.3**
|
||||
|
||||
### Property 4: Neutral signals produce zero LLR
|
||||
|
||||
*For any* valid EvidenceUnit with direction = 0, regardless of all other field values, the computed LLR SHALL be exactly 0.0.
|
||||
|
||||
**Validates: Requirements 1.5, 3.3**
|
||||
|
||||
### Property 5: Effective evidence count n_eff is bounded by cluster size
|
||||
|
||||
*For any* cluster of N signals with non-negative pairwise correlations rho_ij in [0, 1], the computed n_eff SHALL satisfy 0 < n_eff <= N.
|
||||
|
||||
**Validates: Requirements 4.2, 21.4**
|
||||
|
||||
### Property 6: Cluster LLR is clamped to [-2.5, 2.5]
|
||||
|
||||
*For any* cluster configuration with any number of signals and any LLR values, the computed cluster LLR_c SHALL be in the range [-2.5, 2.5].
|
||||
|
||||
**Validates: Requirements 4.4, 4.5**
|
||||
|
||||
### Property 7: Posterior P_up is in open interval (0, 1)
|
||||
|
||||
*For any* set of cluster LLRs (each in [-2.5, 2.5]), any regime evidence multiplier gamma in {0.70, 0.80, 0.90, 1.10}, and any prior P_prior in [0.40, 0.60], the computed P_up SHALL be in (1e-10, 1 - 1e-10).
|
||||
|
||||
**Validates: Requirements 5.3, 21.5**
|
||||
|
||||
### Property 8: Contradiction is zero when evidence is unidirectional
|
||||
|
||||
*For any* set of cluster LLRs where all clusters have the same sign (all positive or all negative), the computed contradiction score SHALL be 0.0.
|
||||
|
||||
**Validates: Requirements 7.7, 21.7**
|
||||
|
||||
### Property 9: Contradiction score is bounded in [0, 1]
|
||||
|
||||
*For any* set of cluster LLRs (including mixed positive and negative), the computed contradiction score SHALL be in the range [0.0, 1.0].
|
||||
|
||||
**Validates: Requirements 7.6**
|
||||
|
||||
### Property 10: Multiplicative confidence is bounded in [0, 1] and suppressed by weak dimensions
|
||||
|
||||
*For any* valid inputs (n_eff_total >= 0, q_values in [0,1], strength in [0,1], regime_confidence_mult in (0,1], contradiction in [0,1], data_quality in [0,1]), the computed confidence SHALL be in [0, 1]. Furthermore, if any single dimension (data_quality, 1-contradiction, or C_quality) is below 0.01, the resulting confidence SHALL be below 0.10.
|
||||
|
||||
**Validates: Requirements 8.3, 8.5, 21.6**
|
||||
|
||||
### Property 11: Fractional Kelly sizing is bounded and respects negative edge
|
||||
|
||||
*For any* valid inputs (P_up in (0,1), b in [1.2, 3.0], confidence in [0,1], data_quality in [0,1], contradiction in [0,1], max_position_pct > 0), the computed portfolio_pct SHALL be in [0, max_position_pct]. When f_kelly = (P_up * b - (1 - P_up)) / b <= 0, portfolio_pct SHALL be exactly 0.
|
||||
|
||||
**Validates: Requirements 14.4, 14.7, 21.8, 21.9**
|
||||
|
||||
### Property 12: Posterior state JSON round-trip
|
||||
|
||||
*For any* valid V3Posterior state (p_up, log_odds, strength, confidence, contradiction, n_eff, data_quality, regime), serializing to JSON and deserializing SHALL produce an equivalent state within floating-point tolerance (1e-10).
|
||||
|
||||
**Validates: Requirements 20.1, 21.10**
|
||||
|
||||
### Property 13: Noisy-OR normalized exposure is bounded in [0, 1]
|
||||
|
||||
*For any* overlap values O_k in [0, 1] for each dimension (geo, supply, commodity, sector) with fixed positive weights, the normalized macro exposure E_macro SHALL be in [0.0, 1.0], reaching exactly 1.0 when all O_k = 1.0.
|
||||
|
||||
**Validates: Requirements 9.1, 9.2**
|
||||
|
||||
### Property 14: Competitive LLR is clamped to [-1.25, 1.25]
|
||||
|
||||
*For any* source LLR, shrunk correlation (non-negative), graph distance (1-3), and pattern confidence in [0,1], the computed competitive LLR SHALL be in [-1.25, 1.25].
|
||||
|
||||
**Validates: Requirements 10.4, 10.5**
|
||||
|
||||
### Property 15: Graph attenuation is zero beyond max distance
|
||||
|
||||
*For any* inputs where graph distance > 3, the computed attenuation SHALL be 0.0, producing zero competitive LLR regardless of other parameters.
|
||||
|
||||
**Validates: Requirements 10.3**
|
||||
|
||||
### Property 16: Projection evidence state decays toward zero
|
||||
|
||||
*For any* initial evidence state A_t and regime decay phi in (0, 1), the projected state A_projected_h = phi^h * A_t SHALL have |A_projected_h| < |A_t| for all h >= 1, converging toward 0 as h increases.
|
||||
|
||||
**Validates: Requirements 11.1, 11.3**
|
||||
|
||||
### Property 17: Data quality score is bounded in [0, 1]
|
||||
|
||||
*For any* valid inputs (extraction_failure_rate in [0,1], extraction_conf_i in [0,1], impact_i in [0,1], age_newest_hours >= 0, N_valid >= 0, N_source_types >= 0), the computed data_quality_score SHALL be in [0, 1].
|
||||
|
||||
**Validates: Requirements 17.6**
|
||||
|
||||
### Property 18: Stop loss is below entry price and take profit is above
|
||||
|
||||
*For any* entry_price > 0, stop_distance_pct in [0.005, 1.0), and reward ratio b >= 1.2, the computed stop_loss SHALL be less than entry_price and take_profit SHALL be greater than entry_price.
|
||||
|
||||
**Validates: Requirements 16.2, 16.3**
|
||||
|
||||
### Property 19: Trailing stop never decreases
|
||||
|
||||
*For any* sequence of current prices and trailing stop computations, each new trailing_stop value SHALL be >= the previous trailing_stop value (monotonically non-decreasing).
|
||||
|
||||
**Validates: Requirements 16.5**
|
||||
|
||||
### Property 20: Regime classification is exhaustive and deterministic
|
||||
|
||||
*For any* valid market data inputs (closing_prices of sufficient length, daily_returns, ATR_20 > 0), the regime classification SHALL produce exactly one of {panic, trend_following, mean_reversion, uncertainty} and the same inputs SHALL always produce the same classification.
|
||||
|
||||
**Validates: Requirements 6.2, 6.3, 6.4, 6.5**
|
||||
|
||||
### Property 21: EvidenceUnit normalization preserves field ranges
|
||||
|
||||
*For any* valid company, macro, or competitive signal input, the normalized EvidenceUnit SHALL have: direction in {-1, 0, +1}, sentiment_strength in [0, 1], impact in [0, 1], extraction_conf in [0, 1], source_cred in [0, 1], novelty in [0, 1], and event_base_rate in (0, 1].
|
||||
|
||||
**Validates: Requirements 1.1, 1.2, 1.3, 1.4, 1.7**
|
||||
|
||||
### Property 22: Portfolio heat rejection is correct
|
||||
|
||||
*For any* set of open positions with stop distances, if the sum of (position_value × stop_distance_pct) exceeds max_portfolio_heat × portfolio_value, then new position entry SHALL be rejected.
|
||||
|
||||
**Validates: Requirements 15.3, 15.5**
|
||||
|
||||
### Property 23: Tier auto-adjustment obeys downgrade-any, upgrade-all logic
|
||||
|
||||
*For any* TierMetrics, if ANY single downgrade condition is met (profit_factor < 1.0 OR drawdown > 0.12 OR calibration_error > 0.20 OR sharpe < 0), the result SHALL be "downgrade". An "upgrade" SHALL only occur when ALL upgrade conditions are simultaneously met.
|
||||
|
||||
**Validates: Requirements 18.3, 18.4**
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Fail-Closed Philosophy
|
||||
|
||||
The v3 engine follows a fail-closed design: when in doubt, suppress the trade rather than emit a false signal.
|
||||
|
||||
| Error Scenario | Response | Fallback |
|
||||
|---|---|---|
|
||||
| `v3_engine_enabled` flag unreadable | Default to heuristic mode | Log warning |
|
||||
| Unhandled exception in v3 pipeline | Fall back to heuristic for that cycle | Log ERROR with traceback, record in metadata |
|
||||
| Missing market data for regime | Default to "uncertainty" regime | Most conservative multipliers |
|
||||
| Missing source statistics | q_source = 0.0 (neutral prior) | Source treated as untrusted |
|
||||
| Missing realized_vol_20d | Use default 0.25 annualized | Conservative volatility estimate |
|
||||
| Division by zero in n_eff | Return n_eff = 1.0 (single signal) | Denominator guard |
|
||||
| NaN/Inf in any computation | Clamp to boundary, log warning | Never propagate NaN to output |
|
||||
| data_quality < 0.50 | Force informational mode | Suppress trade recommendation |
|
||||
| No company evidence (only macro/competitive) | Force informational | Unless macro_only_enabled |
|
||||
|
||||
### Numerical Guards
|
||||
|
||||
All mathematical functions include:
|
||||
- **Sigmoid overflow**: Guard `exp(-x)` for x > 500 or x < -500
|
||||
- **Log domain**: Guard `log(x)` with x > 0 check; `log2(0)` treated as 0 in entropy
|
||||
- **Division by zero**: All denominators checked > 0 before division
|
||||
- **NaN propagation**: All outputs validated with `math.isnan()` check before storage
|
||||
- **Clamp boundaries**: Final values clamped to documented ranges
|
||||
|
||||
### Graceful Degradation Chain
|
||||
|
||||
```
|
||||
v3 pipeline error → heuristic fallback → informational mode → no recommendation
|
||||
```
|
||||
|
||||
Each level preserves audit trail via output metadata.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Dual Testing Approach
|
||||
|
||||
**Property-Based Tests (Hypothesis):**
|
||||
- Library: `hypothesis` (already in use in this project)
|
||||
- Configuration: `@settings(max_examples=100)` minimum per property
|
||||
- File naming: `tests/test_pbt_v3_*.py`
|
||||
- Each property test tagged with: `# Feature: math-core-v3-engine, Property N: <title>`
|
||||
- One property-based test per correctness property (23 properties → 23 PBT tests)
|
||||
|
||||
**Unit Tests (pytest):**
|
||||
- Specific examples with known inputs/outputs for each formula
|
||||
- Edge cases: zero inputs, boundary values, NaN handling
|
||||
- Integration between components (e.g., full pipeline from EvidenceUnit to recommendation)
|
||||
- Error handling paths (DB errors, missing data, feature flag states)
|
||||
|
||||
### Property Test Organization
|
||||
|
||||
| Test File | Properties Covered | Module Under Test |
|
||||
|---|---|---|
|
||||
| `tests/test_pbt_v3_reliability.py` | 1, 2, 3, 4, 21 | scoring.py (q_i, p_correct, LLR) |
|
||||
| `tests/test_pbt_v3_clustering.py` | 5, 6 | worker.py (n_eff, cluster LLR) |
|
||||
| `tests/test_pbt_v3_posterior.py` | 7, 8, 9, 10, 12, 20 | bayesian.py, contradiction.py, worker.py |
|
||||
| `tests/test_pbt_v3_layers.py` | 13, 14, 15 | interpolation.py, signal_propagation.py |
|
||||
| `tests/test_pbt_v3_projection.py` | 16 | projection.py |
|
||||
| `tests/test_pbt_v3_decision.py` | 11, 17, 18, 19, 22 | position_sizer.py, stop_loss_manager.py, eligibility.py |
|
||||
| `tests/test_pbt_v3_tier.py` | 23 | risk/engine.py |
|
||||
|
||||
### Hypothesis Strategy Design
|
||||
|
||||
Key custom strategies for generating valid inputs:
|
||||
|
||||
```python
|
||||
from hypothesis import strategies as st
|
||||
|
||||
# EvidenceUnit generator
|
||||
evidence_units = st.builds(
|
||||
EvidenceUnit,
|
||||
symbol=st.text(min_size=1, max_size=5),
|
||||
layer=st.sampled_from(["company", "macro", "competitive"]),
|
||||
direction=st.sampled_from([-1, 0, 1]),
|
||||
sentiment_strength=st.floats(min_value=0.0, max_value=1.0),
|
||||
impact=st.floats(min_value=0.0, max_value=1.0),
|
||||
extraction_conf=st.floats(min_value=0.0, max_value=1.0),
|
||||
source_cred=st.floats(min_value=0.0, max_value=1.0),
|
||||
novelty=st.floats(min_value=0.0, max_value=1.0),
|
||||
event_base_rate=st.floats(min_value=0.01, max_value=1.0),
|
||||
...
|
||||
)
|
||||
|
||||
# Cluster LLR list generator
|
||||
cluster_llrs = st.lists(
|
||||
st.floats(min_value=-2.5, max_value=2.5),
|
||||
min_size=1, max_size=20,
|
||||
)
|
||||
|
||||
# Regime generator
|
||||
regimes = st.sampled_from(["panic", "trend_following", "mean_reversion", "uncertainty"])
|
||||
```
|
||||
|
||||
### Unit Test Coverage
|
||||
|
||||
| Area | Key Example Tests |
|
||||
|---|---|
|
||||
| EvidenceUnit normalization | Company signal → correct fields; macro → correct horizon mapping |
|
||||
| q_i pipeline | Known inputs → known outputs for each sub-formula |
|
||||
| LLR conversion | p_correct=0.60 → LLR≈0.405; direction=-1 → negative LLR |
|
||||
| Clustering | 3 identical articles → n_eff < 3; independent → n_eff = N |
|
||||
| Posterior | Empty evidence → P_up=0.50; strong bullish → P_up > 0.60 |
|
||||
| Contradiction | All bullish → 0; equal split → high score |
|
||||
| Confidence | Zero data quality → near-zero confidence |
|
||||
| Macro LLR | Full exposure → max LLR ≈ 1.10; zero overlap → LLR ≈ 0 |
|
||||
| Competitive | Distance 4 → zero propagation; direct rival → attenuated signal |
|
||||
| Kelly sizing | p_win=0.3, b=2 → f_kelly < 0 → size = 0 |
|
||||
| Stops | Entry=100, stop_dist=0.02 → stop=98, TP > 100 |
|
||||
| Feature flag | Flag false → heuristic path; flag true → v3 path |
|
||||
| Error fallback | v3 raises → heuristic runs, error logged |
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- Full pipeline: raw signals → EvidenceUnit → q_i → LLR → cluster → posterior → recommendation
|
||||
- Feature flag toggle: verify clean switch between pipelines mid-run
|
||||
- JSONB round-trip: store v3 output in PostgreSQL JSONB, retrieve and verify
|
||||
- Regime transitions: price series that crosses regime boundaries
|
||||
@@ -0,0 +1,314 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
This specification defines the requirements for upgrading the Stonks Oracle signal processing engine from the current dual-mode pipeline (heuristic + probabilistic) to the v3 Calibrated Evidence Engine. The v3 engine replaces arbitrary weighted-sentiment scoring with a principled probabilistic pipeline: calibrated reliability estimation, log-likelihood ratio (LLR) evidence accumulation, correlation-aware clustering, Bayesian posterior assembly, return distribution modeling, and Kelly-criterion position sizing. The upgrade preserves the existing three-layer architecture (company, macro, competitive), the WeightedSignal abstraction, database schema compatibility, and the service boundary structure. The entire v3 engine operates behind a `v3_engine_enabled` feature flag with the heuristic mode retained as a fallback.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **V3_Engine**: The new calibrated evidence engine that replaces the current heuristic and probabilistic scoring modes
|
||||
- **EvidenceUnit**: The canonical normalized shape for all signals (company, macro, competitive) before aggregation
|
||||
- **LLR**: Log-Likelihood Ratio — the calibrated evidence contribution of a single signal or cluster, measured in log-odds units
|
||||
- **Cluster**: A group of correlated signals sharing (symbol, horizon, event_type, source_group, time_bucket)
|
||||
- **n_eff**: Effective evidence count within a cluster after de-correlation adjustment
|
||||
- **Posterior**: The Bayesian posterior probability P_up computed via log-odds accumulation
|
||||
- **Regime**: Market regime classification (panic, trend_following, mean_reversion, uncertainty) derived from EMA trend and volatility indicators
|
||||
- **Contradiction_Score**: A measure of opposing evidence based on LLR entropy and evidence volume
|
||||
- **Data_Quality_Score**: Multiplicative fail-closed quality metric combining parse rate, confidence, freshness, coverage, and diversity
|
||||
- **Fractional_Kelly**: Position sizing method using Kelly criterion scaled by a conservative fraction (0.25) and further modulated by confidence, data quality, and contradiction
|
||||
- **Portfolio_Heat**: Total stop-defined risk dollars across all open positions as a fraction of portfolio value
|
||||
- **Feature_Flag**: The `v3_engine_enabled` runtime toggle that activates the v3 pipeline without code deployment
|
||||
- **Scoring_Service**: The `services/aggregation/scoring.py` module responsible for signal weight computation
|
||||
- **Bayesian_Service**: The `services/aggregation/bayesian.py` module responsible for posterior computation
|
||||
- **Contradiction_Service**: The `services/aggregation/contradiction.py` module responsible for conflict detection
|
||||
- **Regime_Service**: The `services/aggregation/regime.py` module responsible for market regime classification
|
||||
- **Projection_Service**: The `services/aggregation/projection.py` module responsible for trend projection
|
||||
- **Eligibility_Service**: The `services/recommendation/eligibility.py` module responsible for recommendation gating
|
||||
- **Position_Sizer**: The `services/trading/position_sizer.py` module responsible for trade sizing
|
||||
- **Stop_Loss_Manager**: The `services/trading/stop_loss_manager.py` module responsible for stop/TP computation
|
||||
- **Risk_Engine**: The `services/risk/engine.py` module responsible for portfolio risk enforcement
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Canonical Evidence Unit Normalization
|
||||
|
||||
**User Story:** As the aggregation engine, I want all signals normalized into a canonical EvidenceUnit shape, so that company, macro, and competitive signals flow through a single unified pipeline.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHEN a company signal is received, THE V3_Engine SHALL normalize the signal into an EvidenceUnit containing symbol, layer (set to "company"), event_type, source_id, source_group, timestamp, horizon (one of: intraday, 1d, 7d, 30d, 90d), direction (-1/0/+1), sentiment_strength [0,1], impact [0,1], extraction_conf [0,1], source_cred [0,1], novelty [0,1], event_base_rate (float in (0.0, 1.0]), and cluster_id
|
||||
2. WHEN a macro signal is received, THE V3_Engine SHALL normalize the signal into an EvidenceUnit with layer set to "macro", symbol mapped from the macro impact record's ticker, direction mapped from impact_direction (positive→+1, negative→-1, neutral→0), impact mapped from macro_impact_score, source_cred mapped from event confidence, extraction_conf mapped from event confidence, novelty set to 1.0 for new events, source_id mapped from the global event id, source_group set to "macro", and horizon derived from the event's estimated_duration (short_term→7d, medium_term→30d, long_term→90d)
|
||||
3. WHEN a competitive signal is received, THE V3_Engine SHALL normalize the signal into an EvidenceUnit with layer set to "competitive", symbol set to the target ticker, direction mapped from signal_direction (bullish→+1, bearish→-1, neutral→0), impact mapped from signal_strength × relationship_strength, source_cred mapped from pattern_confidence, extraction_conf set to pattern_confidence, novelty set to 1.0, source_id mapped from source_document_id, source_group set to "competitive", and horizon derived from the pattern's time_horizon field
|
||||
4. THE V3_Engine SHALL assign direction value of +1 for signals with sentiment or impact_direction equal to "positive" or "bullish", -1 for "negative" or "bearish", and 0 for "neutral" or "mixed"
|
||||
5. WHEN a signal has direction equal to 0 (neutral), THE V3_Engine SHALL include the signal in quality, coverage, and contradiction context computations but SHALL exclude the signal from directional posterior voting
|
||||
6. IF a required source field (symbol, timestamp, or source_id) is missing or null in the incoming signal, THEN THE V3_Engine SHALL reject the signal, log a warning identifying the signal source and missing field, and not produce an EvidenceUnit for that signal
|
||||
7. IF an optional numeric field (sentiment_strength, impact, extraction_conf, source_cred, novelty) is missing or null, THEN THE V3_Engine SHALL substitute a default value of 0.5 for the missing field
|
||||
8. THE V3_Engine SHALL assign event_base_rate from a configured lookup by event_type, defaulting to 0.10 when no event_type-specific base rate is configured
|
||||
|
||||
### Requirement 2: Calibrated Reliability Computation
|
||||
|
||||
**User Story:** As the scoring engine, I want to compute a calibrated reliability q_i for each signal, so that evidence quality is measured probabilistically instead of via arbitrary weight products.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Scoring_Service SHALL compute extraction reliability as q_ext = sigmoid(k_ext × (extraction_conf - m_ext)) with defaults k_ext = 8.0 and m_ext = 0.55, where extraction_conf is in [0.0, 1.0] and q_ext output is in (0.0, 1.0)
|
||||
2. THE Scoring_Service SHALL compute source reliability using Bayesian shrinkage: E[theta_s] = (alpha_0 + hits_s) / (alpha_0 + beta_0 + hits_s + misses_s) with defaults alpha_0 = 3, beta_0 = 3, and q_source = clamp((E[theta_s] - 0.50) / 0.35, 0.0, 1.0), where hits_s and misses_s are non-negative integers representing the source's historical correct and incorrect directional predictions
|
||||
3. IF a source has zero historical outcomes (hits_s = 0 AND misses_s = 0), THEN THE Scoring_Service SHALL compute q_source = 0.0 from the prior (E[theta_s] = 0.5)
|
||||
4. THE Scoring_Service SHALL compute recency reliability as q_recency = 2^(-age_hours / tau_adaptive) where tau_adaptive = tau_base × (1 + 0.75 × impact + 0.50 × surprise) and surprise = clamp(-log2(event_base_rate) / 5, 0, 1), with age_hours = max((reference_time - signal_timestamp).total_seconds() / 3600, 0.0)
|
||||
5. IF event_base_rate is unavailable or equal to zero, THEN THE Scoring_Service SHALL use a default event_base_rate of 0.10 to prevent undefined logarithm computation
|
||||
6. THE Scoring_Service SHALL use horizon-specific half-life defaults: intraday=2h, 1d=12h, 7d=72h, 30d=240h, 90d=720h
|
||||
7. THE Scoring_Service SHALL compute uniqueness as q_uniqueness = clamp(0.50 + 0.50 × novelty, 0.50, 1.00) × (1 / sqrt(1 + duplicate_count_before)), where duplicate_count_before is the number of other signals in the same cluster that were ingested before this signal
|
||||
8. THE Scoring_Service SHALL compute final signal reliability as q_i = clamp(q_ext × q_source × source_cred × q_recency × q_uniqueness, 0.0, 1.0)
|
||||
9. WHEN q_recency falls below 0.01, THE Scoring_Service SHALL apply a floor of 0.01 only for explainability display output and SHALL use the unmodified q_recency value (including zero) for posterior voting computation
|
||||
|
||||
### Requirement 3: Log-Likelihood Ratio Conversion
|
||||
|
||||
**User Story:** As the posterior engine, I want signals converted to calibrated log-likelihood ratios, so that evidence accumulation follows proper Bayesian updating rules.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Scoring_Service SHALL compute directional correctness probability as p_correct = clamp(0.50 + 0.35 × q_i × impact × sentiment_strength, 0.501, 0.85)
|
||||
2. THE Scoring_Service SHALL compute the signal log-likelihood ratio as LLR_i = direction × ln(p_correct / (1 - p_correct)), where ln denotes the natural logarithm (base e), consistent with the logit function used in posterior assembly
|
||||
3. WHEN direction equals 0 (neutral signal), THE Scoring_Service SHALL produce LLR_i = 0.0, excluding the signal from directional posterior voting while retaining it for quality and contradiction context
|
||||
4. THE Scoring_Service SHALL clamp p_correct to a minimum of 0.501 to ensure LLR_i is always nonzero for directional signals (direction ≠ 0), producing a minimum absolute LLR magnitude of approximately 0.004
|
||||
5. THE Scoring_Service SHALL clamp p_correct to a maximum of 0.85 to prevent any single signal from dominating the posterior, producing a maximum absolute LLR magnitude of approximately 1.735
|
||||
6. FOR ALL valid directional signals (direction ∈ {-1, +1}), THE Scoring_Service SHALL produce LLR_i values with the same sign as direction
|
||||
|
||||
### Requirement 4: Correlation-Aware Evidence Clustering
|
||||
|
||||
**User Story:** As the aggregation engine, I want correlated signals grouped and de-duplicated before posterior assembly, so that near-identical articles cannot inflate evidence counts.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE V3_Engine SHALL cluster signals by (symbol, horizon, event_type, source_group, time_bucket)
|
||||
2. THE V3_Engine SHALL compute effective evidence count as n_eff_c = (sum_i w_i)^2 / (sum_i w_i^2 + 2 × sum_{i<j}(rho_ij × w_i × w_j)) where w_i = abs(LLR_i)
|
||||
3. THE V3_Engine SHALL use default pairwise correlations: rho=0.80 for same wire/story/source group, rho=0.50 for same event different publisher, rho=0.25 for same theme different event, rho=0.00 for independent events
|
||||
4. THE V3_Engine SHALL compute cluster LLR as LLR_c = weighted_mean(LLR_i, abs(LLR_i)) × sqrt(n_eff_c)
|
||||
5. THE V3_Engine SHALL clamp each cluster LLR to the range [-2.5, 2.5] to prevent any single cluster from dominating the posterior
|
||||
|
||||
### Requirement 5: Posterior Assembly via Log-Odds
|
||||
|
||||
**User Story:** As the Bayesian engine, I want to assemble a posterior probability from cluster LLRs and a regime-aware prior, so that the trading decision is based on calibrated belief.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Bayesian_Service SHALL use a neutral base prior of P_prior = 0.50 unless a calibrated symbol/sector prior is stored in the risk_configs table for the given ticker or its sector
|
||||
2. THE Bayesian_Service SHALL compute posterior log-odds as logit(P_up) = logit(P_prior) + sum_c(gamma_regime × LLR_c) where gamma_regime is the regime evidence multiplier from Requirement 6 criterion 6
|
||||
3. THE Bayesian_Service SHALL compute P_up = sigmoid(logit(P_up)) = 1/(1+exp(-logit(P_up))) and P_down = 1 - P_up, clamping P_up to the range [1e-10, 1 - 1e-10] to avoid numerical boundary issues
|
||||
4. THE Bayesian_Service SHALL compute trend strength as strength = abs(2 × P_up - 1), producing a value in [0.0, 1.0] where 0.0 indicates maximum uncertainty and 1.0 indicates maximum directional conviction
|
||||
5. THE Bayesian_Service SHALL apply regime-specific direction thresholds to classify direction from P_up: panic (bullish when P_up >= 0.68, bearish when P_up <= 0.32), trend_following (bullish when P_up >= 0.60, bearish when P_up <= 0.40), mean_reversion (bullish when P_up >= 0.63, bearish when P_up <= 0.37), uncertainty (bullish when P_up >= 0.65, bearish when P_up <= 0.35). WHEN P_up falls between the bullish and bearish thresholds, THE Bayesian_Service SHALL classify direction as neutral
|
||||
6. WHEN P_prior is calibrated from the risk_configs table, THE Bayesian_Service SHALL clamp P_prior to the range [0.40, 0.60] before computing logit(P_prior)
|
||||
7. IF the risk_configs lookup for a calibrated prior fails due to a database error, THEN THE Bayesian_Service SHALL fall back to the neutral base prior of 0.50 and log a warning
|
||||
|
||||
### Requirement 6: Regime Detection v3
|
||||
|
||||
**User Story:** As the regime service, I want to classify market regimes using z-scored indicators and apply regime-appropriate evidence multipliers, so that the engine adapts its sensitivity to market conditions.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Regime_Service SHALL compute trend_z = (EMA_20 - EMA_100) / ATR_20 where EMA_20 and EMA_100 are exponential moving averages of closing prices, and ATR_20 is the 20-day Average True Range. THE Regime_Service SHALL compute vol_ratio = sigma_20 / sigma_100 where sigma_20 and sigma_100 are standard deviations of daily returns
|
||||
2. THE Regime_Service SHALL classify panic when vol_ratio > 1.5 OR abs(trend_z) > 2.5. Panic classification SHALL take priority over all other regimes
|
||||
3. IF the regime is not panic, THEN THE Regime_Service SHALL classify trend_following when abs(trend_z) >= 0.75 AND vol_ratio < 1.3
|
||||
4. IF the regime is neither panic nor trend_following, THEN THE Regime_Service SHALL classify mean_reversion when abs(trend_z) < 0.50 AND vol_ratio < 1.0
|
||||
5. THE Regime_Service SHALL classify uncertainty for all conditions not matching panic, trend_following, or mean_reversion
|
||||
6. THE Regime_Service SHALL apply regime evidence multipliers (gamma_regime): panic=0.70, trend_following=1.10, mean_reversion=0.90, uncertainty=0.80
|
||||
7. THE Regime_Service SHALL apply regime confidence multipliers: panic=0.70, trend_following=1.00, mean_reversion=0.95, uncertainty=0.85
|
||||
8. IF market data is insufficient to compute EMA_100 (fewer than 100 closing prices) or ATR_20 (fewer than 20 bars) or sigma_100 (fewer than 100 daily returns), THEN THE Regime_Service SHALL default to the uncertainty regime
|
||||
|
||||
### Requirement 7: LLR Entropy Contradiction
|
||||
|
||||
**User Story:** As the contradiction service, I want to measure meaningful opposing evidence using LLR entropy, so that contradiction reflects genuine disagreement scaled by evidence volume.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Contradiction_Service SHALL compute E_pos = sum_c(max(LLR_c, 0)) and E_neg = sum_c(max(-LLR_c, 0)) and E_total = E_pos + E_neg
|
||||
2. WHEN E_total equals zero (no directional evidence from any cluster), THE Contradiction_Service SHALL return a contradiction score of 0.0
|
||||
3. WHEN E_total is greater than zero, THE Contradiction_Service SHALL compute f_pos = E_pos / E_total and f_neg = E_neg / E_total where f_pos + f_neg = 1.0
|
||||
4. THE Contradiction_Service SHALL compute H_conflict = -f_pos × log2(f_pos) - f_neg × log2(f_neg), treating 0 × log2(0) as 0 for the boundary case. H_conflict ranges from 0.0 (all one direction) to 1.0 (equal split)
|
||||
5. THE Contradiction_Service SHALL compute volume_factor = 1 - exp(-E_total / 3.0), where 3.0 represents the evidence mass at which contradiction becomes 95% significant
|
||||
6. THE Contradiction_Service SHALL compute the final contradiction score as H_conflict × volume_factor, producing a value in [0.0, 1.0]
|
||||
7. WHEN only one direction of evidence exists (E_pos = 0 or E_neg = 0 but E_total > 0), THE Contradiction_Service SHALL return a contradiction score of 0.0
|
||||
|
||||
### Requirement 8: Multiplicative Confidence v3
|
||||
|
||||
**User Story:** As the aggregation engine, I want confidence computed multiplicatively from independent quality dimensions, so that one weak dimension suppresses the trade rather than being averaged away.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE V3_Engine SHALL compute C_evidence = 1 - exp(-n_eff_total / 5.0) where n_eff_total = sum_c(n_eff_c)
|
||||
2. THE V3_Engine SHALL compute C_quality = weighted_mean(q_i, weight=abs(LLR_i))
|
||||
3. THE V3_Engine SHALL compute confidence = clamp(C_evidence × sqrt(C_quality) × sqrt(max(strength, 0.05)) × regime_confidence_multiplier × (1 - contradiction) × data_quality_score, 0, 1)
|
||||
4. THE V3_Engine SHALL use strength = abs(2 × P_up - 1) as the directional separation term
|
||||
5. WHEN any single confidence dimension is near zero, THE V3_Engine SHALL produce a near-zero final confidence due to the multiplicative formula
|
||||
|
||||
### Requirement 9: Macro Layer v3 (Noisy-OR Exposure)
|
||||
|
||||
**User Story:** As the macro interpolation service, I want to compute normalized exposure via noisy-OR and emit macro evidence as LLR into the shared posterior, so that macro signals integrate with company evidence without special post-hoc modifiers.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE V3_Engine SHALL compute macro exposure as E_raw = 1 - product_k(1 - w_k × O_k) with default weights w_geo=0.35, w_supply=0.25, w_commodity=0.25, w_sector=0.15
|
||||
2. THE V3_Engine SHALL normalize macro exposure as E_macro = E_raw / E_max where E_max = 1 - product_k(1 - w_k)
|
||||
3. THE V3_Engine SHALL apply resilience dampener per market position tier: global_leader=0.70, multinational=0.85, regional=1.00, domestic=1.20
|
||||
4. THE V3_Engine SHALL compute macro likelihood ratio as LLR_macro = macro_direction × log(p_macro / (1 - p_macro)) where p_macro = clamp(0.50 + 0.30 × macro_impact × event_confidence × q_recency, 0.501, 0.80)
|
||||
5. THE V3_Engine SHALL feed macro LLR into the same posterior engine as company evidence without requiring a separate post-hoc modifier
|
||||
|
||||
### Requirement 10: Competitive Layer v3 (Correlation-Shrunk Propagation)
|
||||
|
||||
**User Story:** As the signal propagation service, I want to use correlation-shrunk attenuation for competitive signals, so that propagated evidence is properly discounted by distance and relationship strength.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE V3_Engine SHALL compute shrunk correlation as rho_shrunk = (n / (n + 30)) × rho_rolling + (30 / (n + 30)) × rho_prior with rho_prior_same_sector = 0.30 and rho_prior_cross_sector = 0.10
|
||||
2. THE V3_Engine SHALL compute rho_effective = max(rho_shrunk, 0) to use only positive propagation unless the relationship is explicitly inverse
|
||||
3. THE V3_Engine SHALL compute graph attenuation as attenuation = rho_effective × exp(-0.85 × d_network) with max_distance = 3
|
||||
4. THE V3_Engine SHALL compute competitive LLR as LLR_competitive = LLR_source × attenuation × pattern_confidence
|
||||
5. THE V3_Engine SHALL clamp competitive LLR to the range [-1.25, 1.25] to prevent competitive signals from dominating the posterior
|
||||
|
||||
### Requirement 11: Trend Projection v3 (Posterior State)
|
||||
|
||||
**User Story:** As the projection service, I want to project trends using a posterior state with regime-aware decay, so that projections are grounded in the same Bayesian framework as current estimates.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Projection_Service SHALL maintain an evidence state A_t = phi_regime × A_{t-1} + sum_c(LLR_c), initialized to A_0 = 0.0 when no prior state exists for a ticker-horizon pair
|
||||
2. THE Projection_Service SHALL use regime-specific decay factors: panic phi=0.35, trend_following phi=0.80, mean_reversion phi=0.55, uncertainty phi=0.50
|
||||
3. THE Projection_Service SHALL compute projected alpha as A_projected_h = phi_regime^h × A_t + expected_known_catalyst_LLR_h, where h is the projection horizon in aggregation cycles and expected_known_catalyst_LLR_h defaults to 0.0 when no known catalysts exist
|
||||
4. THE Projection_Service SHALL compute projected probability as P_up_projected_h = sigmoid(logit(P_prior_h) + A_projected_h)
|
||||
5. THE Projection_Service SHALL compute projected strength as abs(2 × P_up_projected_h - 1)
|
||||
6. THE Projection_Service SHALL flag divergence when sign(P_up_projected_h - 0.5) differs from sign(P_up_t - 0.5)
|
||||
7. WHEN market data is insufficient for regime classification, THE Projection_Service SHALL use the uncertainty decay factor (phi=0.50) as the default
|
||||
|
||||
### Requirement 12: Return Distribution and Expected Value Gate
|
||||
|
||||
**User Story:** As the eligibility service, I want to gate recommendations using a return distribution model, so that only trades with positive risk-adjusted expected value pass through.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Eligibility_Service SHALL compute horizon volatility as sigma_h = realized_vol_20d × sqrt(horizon_days / 252), where horizon_days maps to 1 (intraday/1d), 7 (7d), 30 (30d), or 90 (90d)
|
||||
2. THE Eligibility_Service SHALL compute expected return as mu_h = tanh(A_projected_h / 3.0) × confidence × sigma_h
|
||||
3. THE Eligibility_Service SHALL compute EV_long = mu_h - costs - 0.10 × CVaR_5_loss, where costs = spread_cost + slippage_estimate + commission_estimate, and CVaR_5_loss = sigma_h × 1.645 × 1.4 (Gaussian approximation of expected loss beyond the 5th percentile)
|
||||
4. THE Eligibility_Service SHALL compute regime-specific minimum edge: panic=0.0100, trend_following=0.0035, mean_reversion=0.0050, uncertainty=0.0075
|
||||
5. THE Eligibility_Service SHALL require EV_long > min_edge AND EV_long > max(0.0025, 0.25 × costs) for trade eligibility
|
||||
6. THE Eligibility_Service SHALL require confidence >= regime_confidence_min AND contradiction <= regime_contradiction_max AND n_eff_total >= 2.0 AND data_quality_score >= 0.50 for eligibility
|
||||
7. IF realized_vol_20d is unavailable (fewer than 20 trading days of price data), THEN THE Eligibility_Service SHALL use a default volatility of 0.25 (annualized) for sigma_h computation
|
||||
|
||||
### Requirement 13: Recommendation Eligibility and Mode Escalation v3
|
||||
|
||||
**User Story:** As the recommendation service, I want regime-aware eligibility gates and mode escalation, so that recommendation quality matches the rigor of the v3 posterior.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Eligibility_Service SHALL apply regime-specific eligibility thresholds: panic (confidence_min=0.70, contradiction_max=0.25, strength_min=0.36), trend_following (confidence_min=0.55, contradiction_max=0.40, strength_min=0.20), mean_reversion (confidence_min=0.60, contradiction_max=0.35, strength_min=0.26), uncertainty (confidence_min=0.65, contradiction_max=0.30, strength_min=0.30)
|
||||
2. THE Eligibility_Service SHALL map action as BUY when P_up >= bullish_threshold and EV_long > min_edge, SELL when existing position and EV_exit > EV_hold, HOLD when existing position, and WATCH otherwise
|
||||
3. THE Eligibility_Service SHALL escalate to live_eligible when action is BUY or SELL and confidence >= 0.75 and contradiction <= 0.20 and n_eff_total >= 5 and EV_long > 2 × min_edge and risk_engine_passed
|
||||
4. THE Eligibility_Service SHALL escalate to paper_eligible when action is BUY or SELL and confidence >= 0.60 and EV_long > min_edge and risk_engine_passed
|
||||
5. IF eligibility gates are not met, THEN THE Eligibility_Service SHALL assign mode as informational
|
||||
|
||||
### Requirement 14: Fractional Kelly Position Sizing
|
||||
|
||||
**User Story:** As the position sizer, I want to size positions using fractional Kelly criterion, so that position sizes are proportional to edge and constrained by risk.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Position_Sizer SHALL compute stop_distance_pct = max(ATR_pct × ATR_multiplier_regime, sigma_h × 1.25, 0.005) where ATR_pct = ATR_14 / current_price, with regime ATR multipliers: panic=2.5, trend_following=1.8, mean_reversion=1.4, uncertainty=2.0
|
||||
2. THE Position_Sizer SHALL compute reward ratio b = clamp(1.2 + 2.0 × confidence + 1.0 × strength - contradiction, 1.2, 3.0)
|
||||
3. THE Position_Sizer SHALL compute f_kelly = (p_win × b - (1 - p_win)) / b where p_win = P_up from the Bayesian posterior
|
||||
4. THE Position_Sizer SHALL compute final sizing as portfolio_pct = clamp(max(0, f_kelly) × 0.25 × confidence × data_quality_score × (1 - contradiction), 0, max_position_pct)
|
||||
5. THE Position_Sizer SHALL enforce hard caps by reducing portfolio_pct to the minimum of: max_position_pct from the active risk tier, available_sector_capacity_pct, available_correlation_capacity_pct (0 if weighted average absolute correlation with existing positions exceeds 0.80), and available_heat_capacity_pct
|
||||
6. IF portfolio_pct after all caps is less than 0.005 (0.5% of portfolio), THEN THE Position_Sizer SHALL downgrade the recommendation to WATCH with reason "position_below_minimum"
|
||||
7. IF f_kelly is less than or equal to zero, THEN THE Position_Sizer SHALL produce portfolio_pct = 0 and downgrade the recommendation to WATCH with reason "negative_edge"
|
||||
|
||||
### Requirement 15: Stop-Defined Portfolio Heat
|
||||
|
||||
**User Story:** As the risk engine, I want portfolio heat calculated from stop-defined risk dollars, so that risk measurement reflects actual loss exposure rather than position notional.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Risk_Engine SHALL compute risk_dollars = position_value × stop_distance_pct for each open position
|
||||
2. THE Risk_Engine SHALL compute portfolio_heat = sum of risk_dollars across all open positions
|
||||
3. IF portfolio_heat exceeds max_portfolio_heat × portfolio_value, THEN THE Risk_Engine SHALL reject new position entries
|
||||
4. THE Position_Sizer SHALL compute available_heat_capacity = max_portfolio_heat × portfolio_value - current_portfolio_heat
|
||||
5. THE Position_Sizer SHALL reject a position when the new risk_dollars would exceed available_heat_capacity
|
||||
|
||||
### Requirement 16: Regime-Aware Stop Loss and Take Profit
|
||||
|
||||
**User Story:** As the stop loss manager, I want stops and targets computed from regime-aware volatility and dynamic reward ratios, so that exit levels adapt to current market conditions.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Stop_Loss_Manager SHALL compute stop_distance_pct = max(ATR_pct × regime_ATR_multiplier, sigma_h × z_stop, min_stop_pct) with z_stop = 1.25 and min_stop_pct = 0.005
|
||||
2. THE Stop_Loss_Manager SHALL compute stop_loss = entry_price × (1 - stop_distance_pct) for long positions
|
||||
3. THE Stop_Loss_Manager SHALL compute take_profit = entry_price × (1 + b × stop_distance_pct) where b = clamp(1.2 + 2.0 × confidence + 1.0 × strength - contradiction, 1.2, 3.0)
|
||||
4. THE Stop_Loss_Manager SHALL activate trailing stop when unrealized_gain_pct >= 0.50 × take_profit_distance_pct
|
||||
5. THE Stop_Loss_Manager SHALL compute trailing_stop = max(existing_stop, current_price × (1 - trailing_distance_pct)) where trailing_distance_pct = max(ATR_pct × trailing_ATR_mult, sigma_h × 0.75)
|
||||
|
||||
### Requirement 17: Data Quality v3 (Multiplicative Fail-Closed)
|
||||
|
||||
**User Story:** As the suppression layer, I want data quality computed as a multiplicative fail-closed metric, so that a single catastrophic quality failure suppresses the entire recommendation.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE V3_Engine SHALL compute Q_parse = 1 - extraction_failure_rate
|
||||
2. THE V3_Engine SHALL compute Q_conf = weighted_mean(extraction_conf_i, weight=impact_i)
|
||||
3. THE V3_Engine SHALL compute Q_fresh = exp(-age_newest_hours / 168)
|
||||
4. THE V3_Engine SHALL compute Q_coverage = 1 - exp(-N_valid / 5)
|
||||
5. THE V3_Engine SHALL compute Q_diversity = min(1, log2(1 + N_source_types) / log2(4))
|
||||
6. THE V3_Engine SHALL compute data_quality_score = clamp(Q_parse × sqrt(Q_conf) × Q_fresh × Q_coverage × Q_diversity, 0, 1)
|
||||
7. IF data_quality_score < 0.50 OR N_valid < 2 OR Q_parse < 0.50, THEN THE V3_Engine SHALL force the recommendation to informational mode
|
||||
8. IF company evidence is zero and only macro or competitive evidence exists, THEN THE V3_Engine SHALL force the recommendation to informational mode unless macro_only_enabled is configured
|
||||
|
||||
### Requirement 18: Risk Tier Auto-Adjustment v3
|
||||
|
||||
**User Story:** As the risk tier controller, I want tier adjustments based on risk-adjusted performance metrics, so that the engine self-corrects when performance degrades.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE Risk_Engine SHALL track profit_factor_30d (gross_profit / gross_loss over 30 days), max_drawdown_30d (largest peak-to-trough portfolio decline over 30 days as a fraction), calibration_error (mean absolute difference between predicted P_up and realized binary outcome over 30 days), and realized_sharpe_30d (annualized Sharpe ratio of daily returns over 30 days)
|
||||
2. THE Risk_Engine SHALL evaluate tier adjustment conditions once per calendar day after the trading session closes
|
||||
3. THE Risk_Engine SHALL downgrade one tier if any condition is met: profit_factor_30d < 1.0 OR max_drawdown_30d > 0.12 OR calibration_error > 0.20 OR realized_sharpe_30d < 0
|
||||
4. THE Risk_Engine SHALL upgrade one tier only if all conditions are met: profit_factor_30d > 1.35 AND max_drawdown_30d < 0.05 AND calibration_error < 0.12 AND reserve_pool > 0.20 AND N_trades_30d >= 20
|
||||
5. IF a downgrade condition is triggered, THEN THE Risk_Engine SHALL apply the downgrade immediately without waiting for an upgrade evaluation
|
||||
6. THE Risk_Engine SHALL enforce a minimum cooldown of 7 calendar days between consecutive upgrade evaluations to prevent tier oscillation
|
||||
|
||||
### Requirement 19: Feature Flag and Fallback
|
||||
|
||||
**User Story:** As an operator, I want the v3 engine gated behind a runtime feature flag with the heuristic mode as fallback, so that the upgrade can be rolled out safely without downtime.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. WHILE `v3_engine_enabled` is False, THE V3_Engine SHALL route all aggregation through the existing heuristic pipeline without any v3 computation
|
||||
2. WHILE `v3_engine_enabled` is True, THE V3_Engine SHALL route all aggregation through the v3 calibrated evidence pipeline
|
||||
3. THE V3_Engine SHALL read the `v3_engine_enabled` flag from the risk_configs table at the start of each aggregation cycle without requiring a service restart
|
||||
4. IF the v3 pipeline encounters an unhandled error during aggregation, THEN THE V3_Engine SHALL log the error at ERROR level with full traceback and fall back to heuristic mode for that aggregation cycle, recording the fallback event in output metadata
|
||||
5. THE V3_Engine SHALL store a `pipeline_mode` field value of "v3" or "heuristic" in all output records (TrendSummary, Recommendation) JSONB metadata indicating which pipeline produced the result
|
||||
6. IF the `v3_engine_enabled` flag cannot be read from the database (connection error or missing row), THEN THE V3_Engine SHALL default to heuristic mode and log a warning
|
||||
|
||||
### Requirement 20: Database Compatibility and Output Contract
|
||||
|
||||
**User Story:** As the system architect, I want v3 output stored in existing tables using JSONB metadata, so that no schema migration or downtime is required.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. THE V3_Engine SHALL store posterior fields (p_up, log_odds, strength, confidence, contradiction, n_eff, data_quality) in the existing TrendSummary JSONB metadata column
|
||||
2. THE V3_Engine SHALL store return model fields (mu_h, sigma_h, ev_long, min_edge) in the Recommendation JSONB metadata column
|
||||
3. THE V3_Engine SHALL expose an explainability payload containing top_positive_clusters, top_negative_clusters, suppression_reasons, and risk_adjustments
|
||||
4. THE V3_Engine SHALL preserve the existing WeightedSignal abstraction as an intermediate representation before LLR conversion
|
||||
5. THE V3_Engine SHALL preserve all existing database table schemas without requiring new migrations for core functionality
|
||||
|
||||
### Requirement 21: Mathematical Correctness Properties
|
||||
|
||||
**User Story:** As a developer, I want property-based tests validating all v3 mathematical invariants, so that correctness is verified across the input space.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. FOR ALL valid EvidenceUnits, THE V3_Engine SHALL produce q_i values in the range [0, 1]
|
||||
2. FOR ALL valid q_i values, THE V3_Engine SHALL produce p_correct values in the range [0.501, 0.85]
|
||||
3. FOR ALL valid signals with direction != 0, THE V3_Engine SHALL produce LLR values with the same sign as direction
|
||||
4. FOR ALL valid cluster configurations, THE V3_Engine SHALL produce n_eff_c values satisfying 0 < n_eff_c <= N (where N is the cluster size)
|
||||
5. FOR ALL valid cluster LLRs, THE Bayesian_Service SHALL produce P_up values in the range (0, 1) exclusive
|
||||
6. FOR ALL valid inputs, THE V3_Engine SHALL produce confidence values in the range [0, 1]
|
||||
7. FOR ALL valid inputs with no opposing evidence, THE Contradiction_Service SHALL produce a contradiction score of 0
|
||||
8. FOR ALL valid inputs, THE Position_Sizer SHALL produce portfolio_pct values in the range [0, max_position_pct]
|
||||
9. FOR ALL valid inputs where f_kelly <= 0, THE Position_Sizer SHALL produce a portfolio_pct of 0
|
||||
10. FOR ALL valid EvidenceUnit sequences, serializing the posterior state to JSON and deserializing SHALL produce an equivalent posterior state (round-trip property)
|
||||
@@ -0,0 +1,451 @@
|
||||
# Implementation Plan: Math Core v3 Engine
|
||||
|
||||
## Overview
|
||||
|
||||
Incremental upgrade of the Stonks Oracle signal processing engine from dual-mode heuristic/probabilistic to the v3 Calibrated Evidence Engine. Implementation follows the v3 math doc section 20 order: EvidenceUnit → reliability → LLR → clustering → posterior → contradiction → confidence → macro/competitive layers → EV gate → Kelly sizing → stop-defined heat → retire heuristic scoring. All v3 code operates behind the `v3_engine_enabled` feature flag with heuristic fallback.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] 1. EvidenceUnit and LLR conversion behind feature flag
|
||||
- [x] 1.1 Implement EvidenceUnit dataclass and normalization functions in `services/aggregation/scoring.py`
|
||||
- Add frozen dataclass `EvidenceUnit` with all 16 fields (symbol, layer, event_type, source_id, source_group, timestamp, horizon, direction, sentiment_strength, impact, extraction_conf, source_cred, novelty, event_base_rate, cluster_id)
|
||||
- Implement `normalize_company_signal()`, `normalize_macro_signal()`, `normalize_competitive_signal()`
|
||||
- Validate required fields (symbol, timestamp, source_id) — reject with warning on missing
|
||||
- Substitute 0.5 for missing optional numeric fields
|
||||
- Map direction from sentiment/impact_direction strings (+1/-1/0)
|
||||
- Assign event_base_rate from EVENT_TYPE_BASE_RATES lookup (default 0.10)
|
||||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8_
|
||||
|
||||
- [x] 1.2 Implement calibrated reliability pipeline (`compute_v3_reliability`) in `services/aggregation/scoring.py`
|
||||
- Add `SourceStats` dataclass (source_id, hits, misses, alpha_0=3, beta_0=3)
|
||||
- Add `ReliabilityComponents` dataclass (q_ext, q_source, q_recency, q_uniqueness, q_i)
|
||||
- Implement q_ext = sigmoid(8.0 × (extraction_conf - 0.55))
|
||||
- Implement q_source via Bayesian shrinkage: E[theta_s] = (alpha_0 + hits) / (alpha_0 + beta_0 + hits + misses), then clamp((E - 0.50) / 0.35, 0, 1)
|
||||
- Implement q_recency = 2^(-age_hours / tau_adaptive) with adaptive half-life formula
|
||||
- Implement q_uniqueness = clamp(0.5 + 0.5 × novelty, 0.5, 1.0) × (1 / sqrt(1 + dup_count))
|
||||
- Implement q_i = clamp(q_ext × q_source × source_cred × q_recency × q_uniqueness, 0, 1)
|
||||
- Apply floor of 0.01 on q_recency only for explainability display
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9_
|
||||
|
||||
- [x] 1.3 Implement LLR conversion (`compute_llr`) in `services/aggregation/scoring.py`
|
||||
- Compute p_correct = clamp(0.50 + 0.35 × q_i × impact × sentiment_strength, 0.501, 0.85)
|
||||
- Compute LLR_i = direction × ln(p_correct / (1 - p_correct))
|
||||
- Return 0.0 for neutral signals (direction == 0)
|
||||
- Ensure LLR sign always matches direction for directional signals
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_
|
||||
|
||||
- [x] 1.4 Add feature flag routing in `services/aggregation/worker.py`
|
||||
- Read `v3_engine_enabled` from risk_configs table at start of each aggregation cycle
|
||||
- Route to v3 pipeline when True, heuristic when False
|
||||
- Default to heuristic mode if DB read fails (log warning)
|
||||
- Wrap v3 pipeline in try/except — fall back to heuristic on unhandled error (log ERROR with traceback)
|
||||
- Store `pipeline_mode` field ("v3" or "heuristic") in output metadata
|
||||
- _Requirements: 19.1, 19.2, 19.3, 19.4, 19.5, 19.6_
|
||||
|
||||
- [x] 1.5 Write property tests for EvidenceUnit, reliability, and LLR (`tests/test_pbt_v3_reliability.py`)
|
||||
- **Property 1: Reliability q_i is bounded in [0, 1]**
|
||||
- **Property 2: p_correct is bounded in [0.501, 0.85]**
|
||||
- **Property 3: LLR sign matches direction and magnitude is bounded**
|
||||
- **Property 4: Neutral signals produce zero LLR**
|
||||
- **Property 21: EvidenceUnit normalization preserves field ranges**
|
||||
- **Validates: Requirements 1.1–1.8, 2.1–2.9, 3.1–3.6, 21.1–21.3**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 1.6 Write unit tests for EvidenceUnit normalization and LLR conversion (`tests/test_v3_evidence_unit.py`)
|
||||
- Test company signal → EvidenceUnit with correct field mapping
|
||||
- Test macro signal → EvidenceUnit with correct horizon mapping (short_term→7d, medium_term→30d, long_term→90d)
|
||||
- Test competitive signal → EvidenceUnit with correct direction mapping
|
||||
- Test missing required fields → rejection with warning
|
||||
- Test missing optional fields → default 0.5 substitution
|
||||
- Test known inputs through full q_i pipeline → expected outputs
|
||||
- Test LLR boundary cases: p_correct at clamp boundaries
|
||||
- _Requirements: 1.1–1.8, 2.1–2.9, 3.1–3.6_
|
||||
|
||||
- [x] 2. Evidence clustering and n_eff
|
||||
|
||||
- [x] 2.1 Implement correlation-aware clustering in `services/aggregation/worker.py`
|
||||
- Add `EvidenceCluster` dataclass (cluster_id, units, llrs, n_eff, cluster_llr)
|
||||
- Implement `cluster_evidence()` — group EvidenceUnits by (symbol, horizon, event_type, source_group, time_bucket)
|
||||
- Compute cluster_id as hash of grouping key
|
||||
- Define time_bucket resolution per horizon (intraday=1h, 1d=4h, 7d=24h, 30d=72h, 90d=168h)
|
||||
- _Requirements: 4.1_
|
||||
|
||||
- [x] 2.2 Implement n_eff computation in `services/aggregation/worker.py`
|
||||
- Implement `compute_n_eff(llrs, correlations)` using formula: (sum w_i)² / (sum w_i² + 2 × sum_{i<j} rho_ij × w_i × w_j)
|
||||
- Use default pairwise correlations: same wire=0.80, same event diff publisher=0.50, same theme diff event=0.25, independent=0.00
|
||||
- Guard against division by zero (denominator → return n_eff=1.0)
|
||||
- _Requirements: 4.2, 4.3_
|
||||
|
||||
- [x] 2.3 Implement cluster LLR computation in `services/aggregation/worker.py`
|
||||
- Implement `compute_cluster_llr(llrs, n_eff)` = clamp(weighted_mean(LLR_i, |LLR_i|) × sqrt(n_eff), -2.5, 2.5)
|
||||
- Handle single-signal clusters (LLR_c = LLR_i clamped)
|
||||
- Handle all-zero LLR clusters (cluster_llr = 0.0)
|
||||
- _Requirements: 4.4, 4.5_
|
||||
|
||||
- [x] 2.4 Write property tests for clustering (`tests/test_pbt_v3_clustering.py`)
|
||||
- **Property 5: Effective evidence count n_eff is bounded by cluster size**
|
||||
- **Property 6: Cluster LLR is clamped to [-2.5, 2.5]**
|
||||
- **Validates: Requirements 4.2, 4.3, 4.4, 4.5, 21.4**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 2.5 Write unit tests for clustering (`tests/test_v3_clustering.py`)
|
||||
- Test 3 identical articles from same source → n_eff < 3
|
||||
- Test 3 independent articles → n_eff ≈ 3
|
||||
- Test single signal cluster → n_eff = 1.0
|
||||
- Test cluster LLR clamp at ±2.5
|
||||
- Test grouping by correct key dimensions
|
||||
- _Requirements: 4.1–4.5_
|
||||
|
||||
- [x] 3. Checkpoint - Verify foundation layer
|
||||
- Ensure all tests pass for EvidenceUnit, reliability, LLR, and clustering.
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
- [x] 4. Replace trend assembly with posterior P_up
|
||||
|
||||
- [x] 4.1 Implement regime detection v3 in `services/aggregation/regime.py`
|
||||
- Add `V3RegimeClassification` dataclass (regime, trend_z, vol_ratio, evidence_multiplier, confidence_multiplier, phi_decay, atr_multiplier)
|
||||
- Implement `classify_regime_v3(closing_prices, daily_returns, atr_20)` using ATR-normalized trend_z = (EMA_20 - EMA_100) / ATR_20
|
||||
- Compute vol_ratio = sigma_20 / sigma_100
|
||||
- Classification rules in priority: panic (vol_ratio > 1.5 OR |trend_z| > 2.5), trend_following (|trend_z| >= 0.75 AND vol_ratio < 1.3), mean_reversion (|trend_z| < 0.50 AND vol_ratio < 1.0), uncertainty (default)
|
||||
- Assign regime parameters: gamma, confidence_mult, phi, ATR_mult, min_edge
|
||||
- Default to uncertainty when data insufficient
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8_
|
||||
|
||||
- [x] 4.2 Implement posterior assembly via log-odds in `services/aggregation/bayesian.py`
|
||||
- Add `V3Posterior` dataclass (p_up, p_down, log_odds, strength, direction, n_eff_total, regime)
|
||||
- Implement `compute_v3_posterior(clusters, regime, p_prior=0.50)`
|
||||
- Compute logit(P_up) = logit(P_prior) + sum(gamma_regime × LLR_c)
|
||||
- Compute P_up = sigmoid(log_odds), clamp to [1e-10, 1 - 1e-10]
|
||||
- Compute strength = abs(2 × P_up - 1)
|
||||
- Classify direction using regime-specific thresholds (panic: 0.68/0.32, trend_following: 0.60/0.40, mean_reversion: 0.63/0.37, uncertainty: 0.65/0.35)
|
||||
- Load calibrated prior from risk_configs if available (clamp to [0.40, 0.60]), fall back to 0.50 on error
|
||||
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7_
|
||||
|
||||
- [x] 4.3 Write property tests for posterior and regime (`tests/test_pbt_v3_posterior.py`)
|
||||
- **Property 7: Posterior P_up is in open interval (0, 1)**
|
||||
- **Property 20: Regime classification is exhaustive and deterministic**
|
||||
- **Validates: Requirements 5.3, 6.2–6.5, 21.5**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 4.4 Write unit tests for posterior assembly (`tests/test_v3_posterior.py`)
|
||||
- Test empty clusters → P_up = 0.50 (neutral prior)
|
||||
- Test all bullish clusters → P_up > 0.50
|
||||
- Test regime direction thresholds at boundary values
|
||||
- Test prior clamp [0.40, 0.60]
|
||||
- Test regime classification with known inputs
|
||||
- _Requirements: 5.1–5.7, 6.1–6.8_
|
||||
|
||||
- [x] 5. Replace contradiction with LLR entropy
|
||||
|
||||
- [x] 5.1 Implement LLR entropy contradiction in `services/aggregation/contradiction.py`
|
||||
- Add `compute_v3_contradiction(clusters: list[EvidenceCluster]) -> float`
|
||||
- Compute E_pos = sum(max(LLR_c, 0)), E_neg = sum(max(-LLR_c, 0)), E_total = E_pos + E_neg
|
||||
- When E_total == 0 → return 0.0
|
||||
- When only one direction exists (E_pos == 0 or E_neg == 0) → return 0.0
|
||||
- Compute f_pos = E_pos / E_total, f_neg = E_neg / E_total
|
||||
- Compute H_conflict = -f_pos × log2(f_pos) - f_neg × log2(f_neg), treating 0×log2(0) = 0
|
||||
- Compute volume_factor = 1 - exp(-E_total / 3.0)
|
||||
- Return H_conflict × volume_factor, bounded in [0.0, 1.0]
|
||||
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7_
|
||||
|
||||
- [x] 5.2 Write property tests for contradiction (`tests/test_pbt_v3_posterior.py`)
|
||||
- **Property 8: Contradiction is zero when evidence is unidirectional**
|
||||
- **Property 9: Contradiction score is bounded in [0, 1]**
|
||||
- **Validates: Requirements 7.6, 7.7, 21.7**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 5.3 Write unit tests for LLR entropy contradiction (`tests/test_v3_contradiction.py`)
|
||||
- Test all bullish clusters → contradiction = 0.0
|
||||
- Test equal split of evidence → high contradiction near 1.0
|
||||
- Test E_total = 0 → contradiction = 0.0
|
||||
- Test volume_factor growth: small evidence mass → suppressed score
|
||||
- _Requirements: 7.1–7.7_
|
||||
|
||||
- [x] 6. Replace confidence formula
|
||||
|
||||
- [x] 6.1 Implement multiplicative confidence v3 in `services/aggregation/worker.py`
|
||||
- Add `compute_v3_confidence(n_eff_total, q_values, llrs, strength, regime_confidence_mult, contradiction, data_quality) -> float`
|
||||
- Compute C_evidence = 1 - exp(-n_eff_total / 5.0)
|
||||
- Compute C_quality = weighted_mean(q_i, weight=|LLR_i|)
|
||||
- Compute confidence = clamp(C_evidence × sqrt(C_quality) × sqrt(max(strength, 0.05)) × regime_confidence_mult × (1 - contradiction) × data_quality, 0, 1)
|
||||
- _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5_
|
||||
|
||||
- [x] 6.2 Implement data quality v3 in `services/aggregation/worker.py`
|
||||
- Add `compute_v3_data_quality(units, extraction_failure_rate, age_newest_hours, n_source_types) -> float`
|
||||
- Q_parse = 1 - extraction_failure_rate
|
||||
- Q_conf = weighted_mean(extraction_conf_i, weight=impact_i)
|
||||
- Q_fresh = exp(-age_newest_hours / 168)
|
||||
- Q_coverage = 1 - exp(-N_valid / 5)
|
||||
- Q_diversity = min(1, log2(1 + N_source_types) / log2(4))
|
||||
- data_quality_score = clamp(Q_parse × sqrt(Q_conf) × Q_fresh × Q_coverage × Q_diversity, 0, 1)
|
||||
- Force informational mode when data_quality < 0.50 OR N_valid < 2 OR Q_parse < 0.50
|
||||
- Force informational when only macro/competitive evidence unless macro_only_enabled
|
||||
- _Requirements: 17.1, 17.2, 17.3, 17.4, 17.5, 17.6, 17.7, 17.8_
|
||||
|
||||
- [x] 6.3 Write property tests for confidence and data quality (`tests/test_pbt_v3_posterior.py`)
|
||||
- **Property 10: Multiplicative confidence is bounded in [0, 1] and suppressed by weak dimensions**
|
||||
- **Property 17: Data quality score is bounded in [0, 1]**
|
||||
- **Validates: Requirements 8.3, 8.5, 17.6, 21.6**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 6.4 Write unit tests for confidence and data quality (`tests/test_v3_confidence.py`)
|
||||
- Test zero data quality → near-zero confidence
|
||||
- Test full contradiction (1.0) → zero confidence
|
||||
- Test low n_eff → suppressed C_evidence
|
||||
- Test data quality boundary cases (Q_parse < 0.50 → informational)
|
||||
- _Requirements: 8.1–8.5, 17.1–17.8_
|
||||
|
||||
- [x] 7. Checkpoint - Verify core pipeline
|
||||
- Ensure all tests pass for posterior, contradiction, confidence, and data quality.
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
- [x] 8. Convert macro and competitive layers to emit LLR
|
||||
|
||||
- [x] 8.1 Implement noisy-OR macro exposure and LLR emission in `services/aggregation/interpolation.py`
|
||||
- Add `compute_normalized_macro_exposure(overlaps: dict[str, float]) -> float`
|
||||
- E_raw = 1 - product(1 - w_k × O_k) with weights: w_geo=0.35, w_supply=0.25, w_commodity=0.25, w_sector=0.15
|
||||
- E_max = 1 - product(1 - w_k)
|
||||
- E_macro = E_raw / E_max (normalized to [0, 1])
|
||||
- Apply resilience dampener per tier: global_leader=0.70, multinational=0.85, regional=1.00, domestic=1.20
|
||||
- Add `compute_macro_llr(macro_impact, event_confidence, q_recency, macro_direction) -> float`
|
||||
- p_macro = clamp(0.50 + 0.30 × macro_impact × event_confidence × q_recency, 0.501, 0.80)
|
||||
- LLR_macro = macro_direction × ln(p_macro / (1 - p_macro))
|
||||
- Feed macro LLR into shared posterior without separate post-hoc modifier
|
||||
- _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5_
|
||||
|
||||
- [x] 8.2 Implement correlation-shrunk competitive propagation in `services/aggregation/signal_propagation.py`
|
||||
- Add `compute_shrunk_correlation(rho_rolling, n_observations, same_sector) -> float`
|
||||
- rho_prior = 0.30 (same_sector) or 0.10 (cross_sector)
|
||||
- rho_shrunk = (n/(n+30)) × rho_rolling + (30/(n+30)) × rho_prior
|
||||
- rho_effective = max(rho_shrunk, 0)
|
||||
- Add `compute_competitive_llr(llr_source, rho_effective, d_network, pattern_confidence) -> float`
|
||||
- attenuation = rho_effective × exp(-0.85 × d_network), max_distance = 3
|
||||
- LLR_competitive = clamp(llr_source × attenuation × pattern_confidence, -1.25, 1.25)
|
||||
- _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5_
|
||||
|
||||
- [x] 8.3 Write property tests for macro and competitive layers (`tests/test_pbt_v3_layers.py`)
|
||||
- **Property 13: Noisy-OR normalized exposure is bounded in [0, 1]**
|
||||
- **Property 14: Competitive LLR is clamped to [-1.25, 1.25]**
|
||||
- **Property 15: Graph attenuation is zero beyond max distance**
|
||||
- **Validates: Requirements 9.1, 9.2, 10.3, 10.4, 10.5**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 8.4 Write unit tests for macro and competitive layers (`tests/test_v3_layers.py`)
|
||||
- Test noisy-OR: all O_k = 1.0 → E_macro = 1.0; all O_k = 0 → E_macro = 0
|
||||
- Test resilience dampener per tier
|
||||
- Test macro LLR at boundary values
|
||||
- Test shrunk correlation convergence (n → ∞ approaches rho_rolling)
|
||||
- Test competitive LLR clamp at ±1.25
|
||||
- Test distance > 3 → zero attenuation
|
||||
- _Requirements: 9.1–9.5, 10.1–10.5_
|
||||
|
||||
- [x] 9. Replace EV gate with expected-return distribution
|
||||
|
||||
- [x] 9.1 Implement posterior state projection in `services/aggregation/projection.py`
|
||||
- Add `V3ProjectionState` dataclass (a_t, p_up_projected, projected_strength, diverges, phi_regime)
|
||||
- Implement `compute_v3_projection(a_prev, cluster_llrs, regime, p_prior, projection_horizon, known_catalyst_llr=0.0)`
|
||||
- Evidence state: A_t = phi_regime × A_{t-1} + sum(LLR_c), init A_0 = 0.0
|
||||
- Regime decay: panic=0.35, trend_following=0.80, mean_reversion=0.55, uncertainty=0.50
|
||||
- Projected alpha: A_projected = phi^h × A_t + known_catalyst_LLR
|
||||
- P_up_projected = sigmoid(logit(P_prior) + A_projected)
|
||||
- Projected strength = abs(2 × P_up_projected - 1)
|
||||
- Flag divergence when sign(P_up_projected - 0.5) ≠ sign(P_up_t - 0.5)
|
||||
- _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7_
|
||||
|
||||
- [x] 9.2 Implement return distribution and EV gate in `services/recommendation/eligibility.py`
|
||||
- Add `ReturnDistribution` dataclass (sigma_h, mu_h, ev_long, min_edge, eligible)
|
||||
- Add `compute_return_distribution(a_projected, confidence, realized_vol_20d, horizon_days, costs, regime)`
|
||||
- sigma_h = realized_vol_20d × sqrt(horizon_days / 252); default vol = 0.25 if unavailable
|
||||
- mu_h = tanh(A_projected / 3.0) × confidence × sigma_h
|
||||
- CVaR_5 = sigma_h × 1.645 × 1.4
|
||||
- EV_long = mu_h - costs - 0.10 × CVaR_5
|
||||
- Regime min_edge: panic=0.0100, trend_following=0.0035, mean_reversion=0.0050, uncertainty=0.0075
|
||||
- Eligibility: EV_long > min_edge AND EV_long > max(0.0025, 0.25 × costs)
|
||||
- Also require: confidence >= regime_confidence_min, contradiction <= regime_contradiction_max, n_eff_total >= 2.0, data_quality >= 0.50
|
||||
- _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 12.7_
|
||||
|
||||
- [x] 9.3 Implement regime-aware eligibility and mode escalation in `services/recommendation/eligibility.py`
|
||||
- Add regime-specific eligibility thresholds: panic (conf≥0.70, contra≤0.25, str≥0.36), trend_following (conf≥0.55, contra≤0.40, str≥0.20), mean_reversion (conf≥0.60, contra≤0.35, str≥0.26), uncertainty (conf≥0.65, contra≤0.30, str≥0.30)
|
||||
- Action mapping: BUY when P_up >= bullish_threshold and EV > min_edge; SELL when existing position and EV_exit > EV_hold; HOLD when existing; WATCH otherwise
|
||||
- live_eligible: BUY/SELL + conf >= 0.75 + contra <= 0.20 + n_eff >= 5 + EV > 2×min_edge + risk_passed
|
||||
- paper_eligible: BUY/SELL + conf >= 0.60 + EV > min_edge + risk_passed
|
||||
- Otherwise: informational
|
||||
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5_
|
||||
|
||||
- [x] 9.4 Write property tests for projection (`tests/test_pbt_v3_projection.py`)
|
||||
- **Property 16: Projection evidence state decays toward zero**
|
||||
- **Validates: Requirements 11.1, 11.3**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 9.5 Write unit tests for EV gate and eligibility (`tests/test_v3_eligibility.py`)
|
||||
- Test EV_long positive → eligible
|
||||
- Test EV_long negative → ineligible
|
||||
- Test regime-specific min_edge thresholds
|
||||
- Test mode escalation: live vs paper vs informational
|
||||
- Test projection decay convergence
|
||||
- Test divergence flag behavior
|
||||
- _Requirements: 11.1–11.7, 12.1–12.7, 13.1–13.5_
|
||||
|
||||
- [x] 10. Checkpoint - Verify decision layer
|
||||
- Ensure all tests pass for projection, EV gate, eligibility, and layer integrations.
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
- [x] 11. Replace sizing with fractional Kelly under existing risk caps
|
||||
|
||||
- [x] 11.1 Implement fractional Kelly position sizing in `services/trading/position_sizer.py`
|
||||
- Add `compute_kelly_sizing(p_win, b, confidence, data_quality, contradiction, max_position_pct, available_caps) -> float`
|
||||
- Compute reward ratio b = clamp(1.2 + 2.0 × confidence + 1.0 × strength - contradiction, 1.2, 3.0)
|
||||
- Compute f_kelly = (p_win × b - (1 - p_win)) / b
|
||||
- portfolio_pct = clamp(max(0, f_kelly) × 0.25 × confidence × data_quality × (1 - contradiction), 0, max_position_pct)
|
||||
- Apply min of: max_position_pct, sector_capacity, correlation_capacity (0 if avg corr > 0.80), heat_capacity
|
||||
- If portfolio_pct < 0.005 → downgrade to WATCH (reason: position_below_minimum)
|
||||
- If f_kelly <= 0 → portfolio_pct = 0, downgrade to WATCH (reason: negative_edge)
|
||||
- _Requirements: 14.1, 14.2, 14.3, 14.4, 14.5, 14.6, 14.7_
|
||||
|
||||
- [x] 11.2 Implement regime-aware stop loss and take profit in `services/trading/stop_loss_manager.py`
|
||||
- Add v3 stop computation: stop_distance_pct = max(ATR_pct × regime_ATR_mult, sigma_h × 1.25, 0.005)
|
||||
- stop_loss = entry_price × (1 - stop_distance_pct)
|
||||
- take_profit = entry_price × (1 + b × stop_distance_pct) where b = reward ratio
|
||||
- Activate trailing stop when unrealized_gain >= 0.50 × TP distance
|
||||
- trailing_stop = max(existing_stop, current_price × (1 - trailing_distance_pct))
|
||||
- trailing_distance_pct = max(ATR_pct × trailing_ATR_mult, sigma_h × 0.75)
|
||||
- Trailing stop must be monotonically non-decreasing
|
||||
- _Requirements: 16.1, 16.2, 16.3, 16.4, 16.5_
|
||||
|
||||
- [x] 11.3 Write property tests for Kelly sizing and stops (`tests/test_pbt_v3_decision.py`)
|
||||
- **Property 11: Fractional Kelly sizing is bounded and respects negative edge**
|
||||
- **Property 18: Stop loss is below entry price and take profit is above**
|
||||
- **Property 19: Trailing stop never decreases**
|
||||
- **Validates: Requirements 14.4, 14.7, 16.2, 16.3, 16.5, 21.8, 21.9**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 11.4 Write unit tests for Kelly sizing and stops (`tests/test_v3_sizing.py`)
|
||||
- Test p_win=0.3, b=2 → f_kelly < 0 → size = 0
|
||||
- Test p_win=0.7, b=2 → positive size within caps
|
||||
- Test cap enforcement (sector, correlation, heat)
|
||||
- Test position_below_minimum downgrade
|
||||
- Test stop/TP computation with known inputs
|
||||
- Test trailing stop monotonicity over a price sequence
|
||||
- _Requirements: 14.1–14.7, 16.1–16.5_
|
||||
|
||||
- [x] 12. Replace portfolio heat with stop-defined risk dollars
|
||||
|
||||
- [x] 12.1 Implement stop-defined portfolio heat in `services/risk/engine.py`
|
||||
- Add `compute_portfolio_heat(positions, stop_distances) -> float`
|
||||
- risk_dollars = position_value × stop_distance_pct for each position
|
||||
- portfolio_heat = sum(risk_dollars)
|
||||
- Add `check_heat_capacity(current_heat, new_risk_dollars, max_heat_pct, portfolio_value) -> bool`
|
||||
- Reject new entry when current_heat + new_risk_dollars > max_heat_pct × portfolio_value
|
||||
- Integrate available_heat_capacity into Kelly sizing pipeline
|
||||
- _Requirements: 15.1, 15.2, 15.3, 15.4, 15.5_
|
||||
|
||||
- [x] 12.2 Implement risk tier auto-adjustment v3 in `services/risk/engine.py`
|
||||
- Add `TierMetrics` dataclass (profit_factor_30d, max_drawdown_30d, calibration_error, realized_sharpe_30d, n_trades_30d, reserve_pool_pct)
|
||||
- Add `evaluate_tier_adjustment(metrics) -> str` returning 'upgrade'|'downgrade'|'hold'
|
||||
- Downgrade if ANY: profit_factor < 1.0 OR drawdown > 0.12 OR calibration_error > 0.20 OR sharpe < 0
|
||||
- Upgrade only if ALL: profit_factor > 1.35 AND drawdown < 0.05 AND calibration_error < 0.12 AND reserve > 0.20 AND N_trades >= 20
|
||||
- Apply downgrade immediately; enforce 7-day upgrade cooldown
|
||||
- Evaluate once per calendar day after session close
|
||||
- _Requirements: 18.1, 18.2, 18.3, 18.4, 18.5, 18.6_
|
||||
|
||||
- [x] 12.3 Write property tests for heat and tier adjustment (`tests/test_pbt_v3_decision.py`)
|
||||
- **Property 22: Portfolio heat rejection is correct**
|
||||
- **Property 23: Tier auto-adjustment obeys downgrade-any, upgrade-all logic**
|
||||
- **Validates: Requirements 15.3, 15.5, 18.3, 18.4**
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 12.4 Write unit tests for heat and tier (`tests/test_v3_risk.py`)
|
||||
- Test heat computation: 3 positions with known stops → expected heat
|
||||
- Test heat rejection: heat at limit → new entry blocked
|
||||
- Test tier downgrade: single bad metric triggers downgrade
|
||||
- Test tier upgrade: all metrics good → upgrade
|
||||
- Test tier upgrade: one metric bad → hold (not upgrade)
|
||||
- Test 7-day cooldown enforcement
|
||||
- _Requirements: 15.1–15.5, 18.1–18.6_
|
||||
|
||||
- [x] 13. Checkpoint - Verify sizing and risk layer
|
||||
- Ensure all tests pass for Kelly sizing, stops, heat, and tier adjustment.
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
- [x] 14. Retire heuristic scoring to explainability-only mode
|
||||
|
||||
- [x] 14.1 Wire v3 pipeline end-to-end in `services/aggregation/worker.py`
|
||||
- Orchestrate full pipeline: EvidenceUnit → q_i → LLR → cluster → posterior → contradiction → confidence → data_quality
|
||||
- Store v3 posterior in TrendSummary JSONB metadata (p_up, log_odds, strength, confidence, contradiction, n_eff, data_quality, regime)
|
||||
- Store return model in Recommendation JSONB metadata (mu_h, sigma_h, ev_long, min_edge)
|
||||
- Store explainability payload (top_positive_clusters, top_negative_clusters, suppression_reasons, risk_adjustments)
|
||||
- Preserve WeightedSignal as intermediate representation before LLR conversion
|
||||
- _Requirements: 20.1, 20.2, 20.3, 20.4, 20.5_
|
||||
|
||||
- [x] 14.2 Retain heuristic pipeline as fallback with explainability overlay
|
||||
- Keep existing heuristic scoring path fully functional (no removal)
|
||||
- Mark heuristic outputs with `pipeline_mode: "heuristic"` in metadata
|
||||
- Ensure heuristic mode still produces valid TrendSummary and Recommendation objects
|
||||
- Test feature flag toggle: v3 → heuristic → v3 round-trip
|
||||
- _Requirements: 19.1, 19.2, 19.5, 20.4_
|
||||
|
||||
- [x] 14.3 Write property test for JSON round-trip (`tests/test_pbt_v3_posterior.py`)
|
||||
- **Property 12: Posterior state JSON round-trip**
|
||||
- **Validates: Requirements 20.1, 21.10**
|
||||
- Serialize V3Posterior to JSON and deserialize, verify equivalence within 1e-10
|
||||
- Use Hypothesis with `@settings(max_examples=100)`
|
||||
|
||||
- [x] 14.4 Write integration tests for full pipeline (`tests/test_v3_integration.py`)
|
||||
- Test full path: raw signals → EvidenceUnit → q_i → LLR → cluster → posterior → recommendation
|
||||
- Test feature flag false → heuristic path, flag true → v3 path
|
||||
- Test v3 exception → heuristic fallback + error logged
|
||||
- Test output JSONB contains expected v3 fields
|
||||
- _Requirements: 19.1–19.6, 20.1–20.5_
|
||||
|
||||
- [x] 15. Final checkpoint - All tests green
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
- Run full test suite: `.venv/bin/python -m pytest tests/test_pbt_v3_*.py tests/test_v3_*.py -x --tb=short -q`
|
||||
- Verify no regressions in existing heuristic pipeline tests
|
||||
|
||||
## Notes
|
||||
|
||||
- Tasks marked with `*` are optional and can be skipped for faster MVP
|
||||
- Each task references specific requirements for traceability
|
||||
- Checkpoints ensure incremental validation after each logical phase
|
||||
- Property tests validate the 23 correctness properties defined in the design
|
||||
- Unit tests validate specific examples, edge cases, and error handling
|
||||
- The implementation preserves the existing heuristic pipeline as a fully functional fallback
|
||||
- All v3 code is gated behind `v3_engine_enabled` — no changes to production behavior until flag is flipped
|
||||
- Run tests with: `.venv/bin/python -m pytest tests/ -x --tb=short -q`
|
||||
- Property tests use Hypothesis: `@settings(max_examples=100)`
|
||||
|
||||
## Task Dependency Graph
|
||||
|
||||
```json
|
||||
{
|
||||
"waves": [
|
||||
{ "id": 0, "tasks": ["1.1"] },
|
||||
{ "id": 1, "tasks": ["1.2", "1.4"] },
|
||||
{ "id": 2, "tasks": ["1.3"] },
|
||||
{ "id": 3, "tasks": ["1.5", "1.6"] },
|
||||
{ "id": 4, "tasks": ["2.1"] },
|
||||
{ "id": 5, "tasks": ["2.2"] },
|
||||
{ "id": 6, "tasks": ["2.3"] },
|
||||
{ "id": 7, "tasks": ["2.4", "2.5"] },
|
||||
{ "id": 8, "tasks": ["4.1"] },
|
||||
{ "id": 9, "tasks": ["4.2"] },
|
||||
{ "id": 10, "tasks": ["4.3", "4.4"] },
|
||||
{ "id": 11, "tasks": ["5.1"] },
|
||||
{ "id": 12, "tasks": ["5.2", "5.3"] },
|
||||
{ "id": 13, "tasks": ["6.1", "6.2"] },
|
||||
{ "id": 14, "tasks": ["6.3", "6.4"] },
|
||||
{ "id": 15, "tasks": ["8.1", "8.2"] },
|
||||
{ "id": 16, "tasks": ["8.3", "8.4"] },
|
||||
{ "id": 17, "tasks": ["9.1"] },
|
||||
{ "id": 18, "tasks": ["9.2", "9.3"] },
|
||||
{ "id": 19, "tasks": ["9.4", "9.5"] },
|
||||
{ "id": 20, "tasks": ["11.1", "11.2"] },
|
||||
{ "id": 21, "tasks": ["11.3", "11.4"] },
|
||||
{ "id": 22, "tasks": ["12.1", "12.2"] },
|
||||
{ "id": 23, "tasks": ["12.3", "12.4"] },
|
||||
{ "id": 24, "tasks": ["14.1"] },
|
||||
{ "id": 25, "tasks": ["14.2"] },
|
||||
{ "id": 26, "tasks": ["14.3", "14.4"] }
|
||||
]
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user