feat: implement dual-pipeline signal engine service

New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.

Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
  gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading

Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant

Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)

docs/architecture-kubernetes.md

@@ -11,7 +11,7 @@ graph TB
     %% ── External traffic ──────────────────────────────────────────
     internet((Internet))
 
-    subgraph traefik ["kube-system (Traefik Ingress Controller)"]
+    subgraph traefik ["kube-system · Traefik Ingress Controller"]
         direction LR
         ing_dash["stonks.celestium.life"]
         ing_api["stonks-api.celestium.life"]
@@ -28,47 +28,55 @@ graph TB
         direction TB
 
         %% ── API Tier (ingress-facing) ─────────────────────────────
-        subgraph api_tier ["API Tier"]
+        subgraph api_tier ["API Tier · tier: api"]
             direction LR
-            query_api["query-api<br/><i>Deployment (1 replica)</i><br/>:8000"]
-            symbol_registry["symbol-registry<br/><i>Deployment (1 replica)</i><br/>:8000"]
+            query_api["query-api<br/><i>Deployment · 1 replica</i><br/>:8000<br/><i>readiness: /docs</i>"]
+            symbol_registry["symbol-registry<br/><i>Deployment · 1 replica</i><br/>:8000<br/><i>readiness: /docs · liveness: /docs</i>"]
         end
 
         %% ── Frontend Tier ─────────────────────────────────────────
-        subgraph frontend_tier ["Frontend Tier"]
-            dashboard["dashboard<br/><i>Deployment (1 replica)</i><br/>:8080<br/><i>nginx-unprivileged</i>"]
+        subgraph frontend_tier ["Frontend Tier · tier: frontend"]
+            dashboard["dashboard<br/><i>Deployment · 1 replica</i><br/>:8080<br/><i>nginx-unprivileged</i><br/><i>readiness: / · liveness: /</i>"]
         end
 
         %% ── Trading Tier ──────────────────────────────────────────
-        subgraph trading_tier ["Trading Tier"]
+        subgraph trading_tier ["Trading Tier · tier: trading"]
             direction LR
-            trading_engine["trading-engine<br/><i>Deployment (1 replica)</i><br/>:8000"]
-            risk_engine["risk-engine<br/><i>Deployment (1 replica)</i><br/>:8000"]
-            broker_adapter["broker-adapter<br/><i>Deployment (1 replica)</i><br/><i>queue-driven worker</i>"]
+            trading_engine["trading-engine<br/><i>Deployment · 1 replica</i><br/>:8000<br/><i>readiness: /ready · liveness: /health</i>"]
+            risk_engine["risk-engine<br/><i>Deployment · 1 replica</i><br/>:8000"]
+            broker_adapter["broker-adapter<br/><i>Deployment · 1 replica</i><br/><i>queue-driven worker · pipeline-gated</i>"]
         end
 
         %% ── Orchestration Tier ────────────────────────────────────
-        subgraph orchestration_tier ["Orchestration Tier"]
-            scheduler["scheduler<br/><i>Deployment (1 replica)</i><br/><i>runs migrations + seed</i>"]
+        subgraph orchestration_tier ["Orchestration Tier · tier: orchestration"]
+            scheduler["scheduler<br/><i>Deployment · 1 replica · pipeline-gated</i><br/><i>init: migrations → seed → backfill</i>"]
+        end
+
+        %% ── Ingestion Tier ────────────────────────────────────────
+        subgraph ingestion_tier ["Ingestion Tier · tier: ingestion"]
+            ingestion["ingestion<br/><i>Deployment · 1 replica · pipeline-gated</i><br/><i>queue-driven worker</i>"]
         end
 
         %% ── Processing Tier (pipeline workers) ────────────────────
-        subgraph processing_tier ["Processing Tier (pipeline workers)"]
+        subgraph processing_tier ["Processing Tier · tier: processing"]
             direction LR
-            ingestion["ingestion<br/><i>Deployment (2 replicas)</i>"]
-            parser["parser<br/><i>Deployment (2 replicas)</i>"]
-            extractor["extractor<br/><i>Deployment (1 replica)</i>"]
-            aggregation["aggregation<br/><i>Deployment (4 replicas)</i>"]
-            recommendation["recommendation<br/><i>Deployment (1 replica)</i>"]
+            parser["parser<br/><i>Deployment · 2 replicas · pipeline-gated</i>"]
+            extractor["extractor<br/><i>Deployment · 1 replica · pipeline-gated</i>"]
+            aggregation["aggregation<br/><i>Deployment · 4 replicas · pipeline-gated</i>"]
+            recommendation["recommendation<br/><i>Deployment · 1 replica · pipeline-gated</i>"]
         end
 
         %% ── Analytics Tier ────────────────────────────────────────
-        subgraph analytics_tier ["Analytics Tier"]
+        subgraph analytics_tier ["Analytics Tier · tier: analytics"]
             direction LR
-            lake_publisher["lake-publisher<br/><i>Deployment (1 replica)</i><br/><i>queue-driven worker</i>"]
-            hive_metastore["hive-metastore<br/><i>Deployment (1 replica)</i><br/>:9083<br/><i>apache/hive:4.0.0</i>"]
-            trino["trino<br/><i>Deployment (1 replica)</i><br/>:8080<br/><i>trinodb/trino:latest</i>"]
-            superset["superset<br/><i>Deployment (1 replica)</i><br/>:8088<br/><i>custom image</i>"]
+            lake_publisher["lake-publisher<br/><i>Deployment · 1 replica · pipeline-gated</i><br/><i>queue-driven worker</i>"]
+            hive_metastore["hive-metastore<br/><i>Deployment · 1 replica</i><br/>:9083<br/><i>apache/hive:4.0.0</i><br/><i>PVC: hive-metastore-data</i>"]
+            trino["trino<br/><i>Deployment · 1 replica</i><br/>:8080<br/><i>trinodb/trino:latest</i><br/><i>readiness: /v1/info</i>"]
+        end
+
+        %% ── Superset (tier: dashboard in template) ────────────────
+        subgraph superset_block ["Superset · tier: dashboard"]
+            superset["superset<br/><i>Deployment · 1 replica</i><br/>:8088<br/><i>custom image</i><br/><i>PVC: superset-data</i><br/><i>readiness: /health</i>"]
         end
 
         %% ── Helm Secrets ──────────────────────────────────────────
@@ -99,7 +107,7 @@ graph TB
     end
 
     subgraph ollama_ns ["ollama-service namespace"]
-        ollama[("Ollama<br/>ollama:11434<br/><i>GPU: 4070 Ti Super</i>")]
+        ollama[("Ollama<br/>ollama:11434<br/><i>GPU: 4070 Ti Super 16GB</i>")]
     end
 
     %% ── Ingress Routes ────────────────────────────────────────────
@@ -191,6 +199,7 @@ graph TB
     sec_broker -.-> broker_adapter
 
     sec_market -.-> ingestion
+    sec_market -.-> query_api
 
     sec_gmail -.-> trading_engine
 
@@ -216,7 +225,9 @@ graph TB
     classDef tradingSvc fill:#e8a838,stroke:#b07d1a,color:#fff
     classDef processSvc fill:#9b59b6,stroke:#6c3483,color:#fff
     classDef orchSvc fill:#1abc9c,stroke:#148f77,color:#fff
+    classDef ingestionSvc fill:#e67e22,stroke:#bf6516,color:#fff
     classDef analyticsSvc fill:#e74c3c,stroke:#a93226,color:#fff
+    classDef supersetSvc fill:#c0392b,stroke:#96281b,color:#fff
     classDef extSvc fill:#95a5a6,stroke:#717d7e,color:#fff
     classDef secretSvc fill:#f5f5dc,stroke:#999,color:#333
     classDef configSvc fill:#dfe6e9,stroke:#999,color:#333
@@ -225,8 +236,10 @@ graph TB
     class dashboard frontendSvc
     class trading_engine,risk_engine,broker_adapter tradingSvc
     class scheduler orchSvc
-    class ingestion,parser,extractor,aggregation,recommendation processSvc
-    class lake_publisher,hive_metastore,trino,superset analyticsSvc
+    class ingestion ingestionSvc
+    class parser,extractor,aggregation,recommendation processSvc
+    class lake_publisher,hive_metastore,trino analyticsSvc
+    class superset supersetSvc
     class postgres,redis,minio,ollama extSvc
     class sec_core,sec_broker,sec_market,sec_gmail,sec_dashboard secretSvc
     class configmap configSvc
@@ -284,8 +297,8 @@ The following services have **no inbound network policy** — they are queue-dri
 
 | Service | Tier | Behavior |
 |---------|------|----------|
-| scheduler | orchestration | Polls DB, enqueues to Redis |
-| ingestion | processing | Reads from `stonks:queue:ingestion`, writes to DB/MinIO/Redis |
+| scheduler | orchestration | Polls DB, enqueues to Redis. Runs migrations + seed + backfill as init containers |
+| ingestion | ingestion | Reads from `stonks:queue:ingestion`, writes to DB/MinIO/Redis. Egress to Polygon.io/News APIs |
 | parser | processing | Reads from `stonks:queue:parsing`, writes to DB/Redis |
 | extractor | processing | Reads from `stonks:queue:extraction`, calls Ollama, writes to DB/Redis |
 | aggregation | processing | Reads from `stonks:queue:aggregation`, writes to DB/Redis |
@@ -294,22 +307,24 @@ The following services have **no inbound network policy** — they are queue-dri
 
 ## Service Tier Summary
 
-| Tier | Services | Ingress? | Replicas | Notes |
-|------|----------|----------|----------|-------|
-| **api** | query-api, symbol-registry | Yes (Traefik) | 1 each | FastAPI, readiness probes on `/docs` |
-| **frontend** | dashboard | Yes (Traefik) | 1 | nginx-unprivileged on :8080, proxies to API services |
-| **trading** | trading-engine, risk-engine, broker-adapter | trading-engine: Yes; risk-engine: internal only; broker-adapter: denied | 1 each | trading-engine has egress to Alpaca + Gmail |
-| **orchestration** | scheduler | No | 1 | Runs DB migrations + seed as init containers |
-| **processing** | ingestion, parser, extractor, aggregation, recommendation | No | 2, 2, 1, 4, 1 | Pipeline-gated by `pipelineEnabled` toggle |
-| **analytics** | lake-publisher, trino, hive-metastore, superset | trino + superset: Yes; others: No | 1 each | lake-publisher is pipeline-gated |
+| Tier | Services | Ingress? | Replicas | Pipeline-Gated? | Notes |
+|------|----------|----------|----------|-----------------|-------|
+| **api** | query-api, symbol-registry | Yes (Traefik) | 1 each | No | FastAPI, readiness probes on `/docs` |
+| **frontend** | dashboard | Yes (Traefik) | 1 | No | nginx-unprivileged on :8080, proxies to API services |
+| **trading** | trading-engine, risk-engine, broker-adapter | trading-engine: Yes; risk-engine: internal only; broker-adapter: denied | 1 each | broker-adapter only | trading-engine has egress to Alpaca + Gmail |
+| **orchestration** | scheduler | No | 1 | Yes | Runs DB migrations + seed + backfill as init containers |
+| **ingestion** | ingestion | No | 1 | Yes | Fetches from external APIs (Polygon.io, news, filings) |
+| **processing** | parser, extractor, aggregation, recommendation | No | 2, 1, 4, 1 | Yes | Queue-driven pipeline workers |
+| **analytics** | lake-publisher, trino, hive-metastore | trino: Yes (Traefik); others: No | 1 each | lake-publisher only | trino + hive-metastore gated by `trino.enabled` / `hiveMetastore.enabled` |
+| **dashboard** (Superset) | superset | Yes (Traefik) | 1 | No | Gated by `superset.enabled`, custom image with trino + psycopg2 drivers |
 
 ## Secret Consumption Map
 
 | Secret | Keys | Consumers |
 |--------|------|-----------|
-| `stonks-core-secrets` | POSTGRES_PASSWORD, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, REDIS_PASSWORD | All 13 app services + hive-metastore, trino, superset |
+| `stonks-core-secrets` | POSTGRES_PASSWORD, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, REDIS_PASSWORD | All 13 app services + hive-metastore (init), trino (init), superset |
 | `stonks-broker-secrets` | BROKER_API_KEY, BROKER_API_SECRET, BROKER_BASE_URL | ingestion, trading-engine, risk-engine, broker-adapter |
-| `stonks-market-secrets` | MARKET_DATA_API_KEY | ingestion |
+| `stonks-market-secrets` | MARKET_DATA_API_KEY | ingestion, query-api |
 | `stonks-gmail-secrets` | GMAIL_SENDER, GMAIL_RECIPIENT, GMAIL_APP_PASSWORD | trading-engine |
 | `stonks-dashboard-secrets` | SUPERSET_SECRET_KEY, SUPERSET_ADMIN_PASSWORD | superset |
 
@@ -336,10 +351,10 @@ These services run outside the `stonks-oracle` namespace and are referenced via
 
 The analytics stack runs within the `stonks-oracle` namespace:
 
-1. **Lake Publisher** writes Parquet fact tables to MinIO at `s3a://stonks-lakehouse/warehouse`
-2. **Hive Metastore** (Apache Hive 4.0.0) manages table metadata, backed by embedded Derby DB with a PVC for persistence. Connects to MinIO for S3A filesystem access.
-3. **Trino** queries the lakehouse via Hive Metastore (thrift://hive-metastore:9083). Exposes two catalogs: `lakehouse` (Hive connector) and `iceberg` (Iceberg connector). Both connect to MinIO for data access.
-4. **Superset** connects to Trino for lakehouse queries and to PostgreSQL for its metadata DB. Uses Redis for caching. Exposed externally via Traefik ingress.
+1. **Lake Publisher** writes Parquet fact tables to MinIO at `s3a://stonks-lakehouse/warehouse`. Pipeline-gated — scales to 0 when `pipelineEnabled: false`.
+2. **Hive Metastore** (Apache Hive 4.0.0) manages table metadata, backed by embedded Derby DB with a PVC (`hive-metastore-data`) for persistence. Connects to MinIO for S3A filesystem access. Gated by `hiveMetastore.enabled`.
+3. **Trino** queries the lakehouse via Hive Metastore (`thrift://hive-metastore:9083`). Exposes two catalogs: `lakehouse` (Hive connector) and `iceberg` (Iceberg connector). Both connect to MinIO for data access. Gated by `trino.enabled`. Readiness probe on `/v1/info`.
+4. **Superset** connects to Trino for lakehouse queries and to PostgreSQL for its metadata DB. Uses Redis for caching. Exposed externally via Traefik ingress. Gated by `superset.enabled`. Uses custom image (`registry.celestium.life/stonks-oracle/superset:latest`) with trino + psycopg2 drivers. PVC (`superset-data`) for persistence.
 
 ## Ingress Routes
 
@@ -353,3 +368,13 @@ All ingress resources use the `traefik` IngressClass with TLS certificates issue
 | `stonks-trading.celestium.life` | trading-engine | 8000 | `stonks-trading-tls` |
 | `stonks-dash.celestium.life` | superset | 8088 | `stonks-dash-tls` |
 | `stonks-trino.celestium.life` | trino | 8080 | `stonks-trino-tls` |
+
+## Deployment Stages
+
+The Helm chart supports multiple deployment stages via value override files:
+
+| Stage | Override File | Namespace | Key Differences |
+|-------|--------------|-----------|-----------------|
+| **Production** | `values.yaml` (base) | `stonks-oracle` | Full analytics stack, all services |
+| **Paper** | `values-paper.yaml` | `stonks-oracle` | `BROKER_MODE=paper`, `DEPLOY_STAGE=paper`, separate DB (`stonks_paper`), Redis DB 2, paper-specific ingress hostnames |
+| **Beta** | `values-beta.yaml` | `stonks-oracle-beta` | `DEPLOY_STAGE=beta`, `LOG_LEVEL=DEBUG`, separate DB (`stonks_beta`), Redis DB 1, analytics stack disabled, beta-specific ingress hostnames |