feat: implement dual-pipeline signal engine service

New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.

Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
  gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading

Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant

Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)

docs/architecture-data-pipeline.md

@@ -18,13 +18,13 @@ flowchart TB
     end
 
     %% ── Scheduler ─────────────────────────────────────────────────
-    scheduler["<b>Scheduler</b><br/><i>services.scheduler.app</i><br/>Cadence polling, rate limiting,<br/>backoff &amp; stale recovery"]
+    scheduler["<b>Scheduler</b><br/><i>services.scheduler.app</i><br/>Cadence polling, rate limiting,<br/>backoff, stale recovery,<br/>periodic aggregation,<br/>report scheduling"]
 
     sources -.->|"API polling<br/>on cadence"| scheduler
 
     %% ── Ingestion Queue ───────────────────────────────────────────
     q_ingestion[["stonks:queue:ingestion"]]
-    scheduler -->|"rpush job"| q_ingestion
+    scheduler -->|"rpush job<br/>(company, macro,<br/>global market)"| q_ingestion
 
     %% ── Ingestion Worker ──────────────────────────────────────────
     ingestion["<b>Ingestion</b><br/><i>services.ingestion.worker</i><br/>Adapter dispatch, dedupe,<br/>raw artifact upload"]
@@ -42,7 +42,7 @@ flowchart TB
 
     %% ── Parsing Queue ─────────────────────────────────────────────
     q_parsing[["stonks:queue:parsing"]]
-    ingestion -->|"rpush<br/>(news, filings,<br/>web_scrape)"| q_parsing
+    ingestion -->|"rpush<br/>(news, filings,<br/>web_scrape, macro)"| q_parsing
 
     %% ── Parser Worker ─────────────────────────────────────────────
     parser["<b>Parser</b><br/><i>services.parser.worker</i><br/>HTML parsing, quality scoring,<br/>company mention detection"]
@@ -50,7 +50,7 @@ flowchart TB
     q_parsing -->|"lpop"| parser
 
     minio_norm[("MinIO<br/><i>Normalized Text</i><br/><i>Parser Output JSON</i>")]
-    parser -->|"upload normalized text"| minio_norm
+    parser -->|"upload normalized text<br/>+ structured output"| minio_norm
     parser -->|"update document status,<br/>insert mentions"| pg_docs
 ```
 
@@ -70,18 +70,23 @@ flowchart TB
     parser -->|"rpush<br/>(standard docs)"| q_extraction
     parser -->|"rpush<br/>(macro_event docs)"| q_macro
 
+    %% ── Scheduler Recovery ────────────────────────────────────────
+    scheduler_recovery(("Scheduler<br/><i>stale recovery &amp;<br/>failed retry</i>"))
+    scheduler_recovery -.->|"re-enqueue orphaned<br/>parsed docs"| q_extraction
+    scheduler_recovery -.->|"re-enqueue orphaned<br/>macro docs"| q_macro
+
     %% ── Extractor Worker ──────────────────────────────────────────
     subgraph extractor_svc ["Extractor Service"]
         direction TB
-        ext_main["<b>Extractor</b><br/><i>services.extractor.main</i><br/>Alternates between queues<br/>(2 extraction : 1 macro)"]
+        ext_main["<b>Extractor</b><br/><i>services.extractor.main</i><br/>Alternates between queues<br/>(2 extraction : 1 macro)<br/>Token budget enforcement"]
     end
 
     q_extraction -->|"lpop"| ext_main
     q_macro -->|"lpop"| ext_main
 
     %% ── Ollama LLM ───────────────────────────────────────────────
-    ollama["<b>Ollama</b><br/><i>LLM Inference</i><br/>document-extractor agent<br/>event-classifier agent"]
-    ext_main <-->|"HTTP /api/generate"| ollama
+    ollama["<b>Ollama / vLLM</b><br/><i>LLM Inference</i><br/>document-extractor agent<br/>event-classifier agent"]
+    ext_main <-->|"HTTP /api/generate<br/>(AgentConfigResolver<br/>selects model + variant)"| ollama
 
     %% ── Signal Layer 1: Company ───────────────────────────────────
     subgraph layer1 ["Layer 1 — Company Signals"]
@@ -95,7 +100,7 @@ flowchart TB
     subgraph layer2 ["Layer 2 — Macro Signals"]
         direction LR
         ge["global_events"]
-        mir["macro_impact_records<br/><i>per-company interpolation</i>"]
+        mir["macro_impact_records<br/><i>per-company interpolation<br/>via exposure profiles</i>"]
         ge --> mir
     end
 
@@ -106,6 +111,10 @@ flowchart TB
     q_agg[["stonks:queue:aggregation"]]
     ext_main -->|"rpush<br/>(per ticker)"| q_agg
 
+    %% ── Scheduler Periodic Aggregation ────────────────────────────
+    scheduler_agg(("Scheduler<br/><i>periodic aggregation<br/>every ~15 min</i>"))
+    scheduler_agg -.->|"rpush all<br/>active tickers"| q_agg
+
     %% ── Aggregation Worker ────────────────────────────────────────
     aggregation["<b>Aggregation</b><br/><i>services.aggregation.main</i><br/>Trend windows, scoring,<br/>contradiction detection"]
 
@@ -133,6 +142,8 @@ flowchart TB
 
 ## Recommendation → Trading → Broker
 
+The recommendation worker consumes from the recommendation queue. The trading engine does **not** consume from a queue — it polls the `recommendations` table in PostgreSQL on a configurable interval, evaluates each recommendation through its decision pipeline, and pushes "act" decisions to the broker queue.
+
 ```mermaid
 flowchart TB
     %% ── Recommendation Queue ──────────────────────────────────────
@@ -144,19 +155,23 @@ flowchart TB
 
     q_rec -->|"lpop"| recommendation
 
-    ollama_thesis["<b>Ollama</b><br/><i>thesis-rewriter agent</i><br/>(optional LLM rewrite)"]
+    ollama_thesis["<b>Ollama / vLLM</b><br/><i>thesis-rewriter agent</i><br/>(AgentConfigResolver<br/>selects model + variant)"]
     recommendation <-->|"rewrite thesis<br/>(trading-eligible only)"| ollama_thesis
 
     pg_recs[("PostgreSQL<br/><i>recommendations,<br/>recommendation_evidence,<br/>risk_evaluations</i>")]
     recommendation -->|"persist recommendation<br/>+ evidence + risk eval"| pg_recs
 
+    %% ── Lake Publication (inline) ─────────────────────────────────
+    minio_rec_lake[("MinIO<br/><i>Lakehouse</i><br/>recommendation facts")]
+    recommendation -->|"publish_recommendation_facts<br/>(Parquet)"| minio_rec_lake
+
     %% ── Trading Engine ────────────────────────────────────────────
     subgraph trading_loop ["Trading Engine Decision Loop"]
         direction TB
         poll["Poll recommendations<br/><i>action IN (buy, sell)<br/>mode IN (paper, live)<br/>generated_at &gt; last_poll</i>"]
         dedup_check["Redis dedup check<br/><i>stonks:dedupe:trading:*</i>"]
-        evaluate["evaluate_recommendation<br/><i>Circuit breaker check<br/>Trading window check<br/>Confidence gate<br/>Sector exposure check<br/>Correlation check<br/>Earnings blackout</i>"]
-        size["Position sizing<br/><i>Kelly criterion,<br/>risk tier limits</i>"]
+        evaluate["evaluate_recommendation<br/><i>Circuit breaker check<br/>Trading window check<br/>Confidence gate<br/>Sector exposure check<br/>Correlation check<br/>Earnings blackout<br/>Max positions check</i>"]
+        size["Position sizing<br/><i>Kelly criterion,<br/>risk tier limits,<br/>micro-trade support</i>"]
         decide{{"Decision"}}
         poll --> dedup_check --> evaluate --> size --> decide
     end
@@ -170,22 +185,30 @@ flowchart TB
 
     pg_decisions[("PostgreSQL<br/><i>trading_decisions</i>")]
 
+    %% ── Manual Override ───────────────────────────────────────────
+    trading_api(("Trading API<br/><i>POST /override/order</i>"))
+    trading_api -->|"rpush<br/>manual order"| q_broker
+
     %% ── Broker Adapter ────────────────────────────────────────────
-    broker["<b>Broker Adapter</b><br/><i>services.adapters.broker_service</i><br/>Risk evaluation, idempotency,<br/>order submission, fill tracking"]
+    broker["<b>Broker Adapter</b><br/><i>services.adapters.broker_service</i><br/>Idempotency, risk evaluation,<br/>approval gate, order submission,<br/>fill tracking, position sync"]
 
     q_broker -->|"lpop"| broker
 
     %% ── Risk Engine ───────────────────────────────────────────────
-    risk["<b>Risk Engine</b><br/><i>services.risk.app</i><br/>POST /evaluate<br/>Approval workflow"]
-    broker <-->|"evaluate order"| risk
+    risk["<b>Risk Engine</b><br/><i>services.risk.app</i><br/>evaluate_order()<br/>Position limits, sector exposure,<br/>daily loss caps, approval workflow"]
+    broker -->|"evaluate order<br/>(inline call)"| risk
 
     %% ── Alpaca ────────────────────────────────────────────────────
-    alpaca["<b>Alpaca</b><br/><i>Paper Trading API</i><br/>Order submission,<br/>position sync"]
-    broker <-->|"submit order /<br/>sync positions"| alpaca
+    alpaca["<b>Alpaca</b><br/><i>Paper Trading API</i><br/>Order submission,<br/>position sync,<br/>account state"]
+    broker <-->|"submit order /<br/>sync positions /<br/>sync order status"| alpaca
 
-    pg_orders[("PostgreSQL<br/><i>orders, order_events,<br/>positions,<br/>portfolio_snapshots</i>")]
+    pg_orders[("PostgreSQL<br/><i>orders, order_events,<br/>positions,<br/>portfolio_snapshots,<br/>broker_accounts</i>")]
     broker -->|"persist order,<br/>events, positions"| pg_orders
 
+    %% ── Lake Publication (broker inline) ──────────────────────────
+    minio_broker_lake[("MinIO<br/><i>Lakehouse</i><br/>order + fill + position facts")]
+    broker -->|"publish_trade_order<br/>publish_trade_fill<br/>publish_positions_daily_batch<br/>(Parquet)"| minio_broker_lake
+
     %% ── Notifications ─────────────────────────────────────────────
     subgraph notifications ["Notifications"]
         direction LR
@@ -198,28 +221,32 @@ flowchart TB
 
 ## Analytical Branch — Lake Publisher
 
-The lake publisher runs as a separate worker, consuming from its own queue and writing partitioned Parquet fact tables to MinIO for analytical queries.
+The lake publisher runs as a separate worker, consuming from its own queue and writing partitioned Parquet fact tables to MinIO for analytical queries. Some services (broker adapter, recommendation worker) also publish facts directly to MinIO inline, bypassing the queue.
 
 ```mermaid
 flowchart LR
     %% ── Lake Publish Queue ────────────────────────────────────────
     q_lake[["stonks:queue:lake_publish"]]
 
-    various(("Various Services<br/><i>ingestion, extractor,<br/>recommendation,<br/>broker adapter</i>"))
-    various -->|"enqueue_lake_job"| q_lake
+    various(("Upstream Services<br/><i>via enqueue_lake_job()</i>"))
+    various -->|"rpush job<br/>(job_type + entity_id)"| q_lake
 
     %% ── Lake Publisher Worker ─────────────────────────────────────
-    lake["<b>Lake Publisher</b><br/><i>services.lake_publisher.jobs</i><br/>Transforms operational data<br/>into analytical facts"]
+    lake["<b>Lake Publisher</b><br/><i>services.lake_publisher.jobs</i><br/>Transforms operational data<br/>into analytical facts<br/><i>15 job types supported</i>"]
 
     q_lake -->|"lpop"| lake
 
-    pg_source[("PostgreSQL<br/><i>Operational Tables</i><br/>documents, extractions,<br/>orders, positions, events")]
+    pg_source[("PostgreSQL<br/><i>Operational Tables</i><br/>documents, extractions,<br/>orders, positions, events,<br/>global_events, macro_impacts,<br/>competitive_signals")]
     lake -->|"query source data"| pg_source
 
     %% ── MinIO Parquet ─────────────────────────────────────────────
     minio_lake[("MinIO<br/><i>Lakehouse Bucket</i><br/>Partitioned Parquet<br/>/year=/month=/day=")]
     lake -->|"write Parquet files"| minio_lake
 
+    %% ── Inline Publishers ─────────────────────────────────────────
+    inline(("Inline Publishers<br/><i>broker adapter,<br/>recommendation worker</i>"))
+    inline -->|"publish_* functions<br/>(direct Parquet write)"| minio_lake
+
     %% ── Trino ─────────────────────────────────────────────────────
     trino["<b>Trino</b><br/><i>SQL Query Engine</i><br/>Hive connector → MinIO"]
     minio_lake -->|"read via<br/>Hive Metastore"| trino
@@ -238,18 +265,40 @@ flowchart LR
     query_api --> dashboard
 ```
 
+## Report Generation
+
+The scheduler manages report generation as a sub-loop, enqueuing daily and weekly report jobs to a dedicated queue and consuming them inline.
+
+```mermaid
+flowchart LR
+    scheduler["<b>Scheduler</b><br/><i>report schedule check</i><br/>daily @ 16:30 ET<br/>weekly @ Saturday"]
+
+    q_report[["stonks:queue:report_generation"]]
+    scheduler -->|"rpush<br/>(daily/weekly)"| q_report
+
+    scheduler_consumer["<b>Scheduler</b><br/><i>report consumer loop</i><br/>pops up to 5 jobs/cycle"]
+    q_report -->|"lpop"| scheduler_consumer
+
+    generator["<b>Report Generator</b><br/><i>services.reporting.generator</i>"]
+    scheduler_consumer -->|"process_report_job()"| generator
+
+    pg_reports[("PostgreSQL<br/><i>trading_reports</i>")]
+    generator -->|"persist report"| pg_reports
+```
+
 ## Complete Queue Topology
 
 | Queue | Full Key | Producer(s) | Consumer |
 |-------|----------|-------------|----------|
-| Ingestion | `stonks:queue:ingestion` | Scheduler | Ingestion Worker |
-| Parsing | `stonks:queue:parsing` | Ingestion Worker | Parser Worker |
-| Extraction | `stonks:queue:extraction` | Parser (standard docs) | Extractor Worker |
-| Macro Classification | `stonks:queue:macro_classification` | Parser (macro_event docs), Scheduler | Extractor Worker |
-| Aggregation | `stonks:queue:aggregation` | Extractor Worker | Aggregation Worker |
-| Recommendation | `stonks:queue:recommendation` | Aggregation Worker | Recommendation Worker |
-| Broker Orders | `stonks:queue:broker_orders` | Trading Engine, Trading API (manual overrides) | Broker Adapter |
-| Lake Publish | `stonks:queue:lake_publish` | Various services | Lake Publisher |
+| Ingestion | `stonks:queue:ingestion` | Scheduler (company, macro, global market sources) | Ingestion Worker |
+| Parsing | `stonks:queue:parsing` | Ingestion Worker (news, filings, web_scrape, macro) | Parser Worker |
+| Extraction | `stonks:queue:extraction` | Parser (standard docs), Scheduler (stale recovery) | Extractor Worker |
+| Macro Classification | `stonks:queue:macro_classification` | Parser (macro_event docs), Scheduler (stale/failed recovery) | Extractor Worker |
+| Aggregation | `stonks:queue:aggregation` | Extractor Worker (per ticker), Scheduler (periodic, all tickers) | Aggregation Worker |
+| Recommendation | `stonks:queue:recommendation` | Aggregation Worker (ticker + window, 5 min dedup TTL) | Recommendation Worker |
+| Broker Orders | `stonks:queue:broker_orders` | Trading Engine (act decisions), Trading API (manual overrides) | Broker Adapter |
+| Lake Publish | `stonks:queue:lake_publish` | Various services (via `enqueue_lake_job()`) | Lake Publisher |
+| Report Generation | `stonks:queue:report_generation` | Scheduler (daily/weekly triggers) | Scheduler (inline consumer) |
 
 Dead-letter queues follow the pattern `stonks:dlq:<queue_name>` and are populated when a job exhausts its retry budget.
 
@@ -257,18 +306,25 @@ Dead-letter queues follow the pattern `stonks:dlq:<queue_name>` and are populate
 
 | Store | Role | Key Tables / Buckets |
 |-------|------|---------------------|
-| **PostgreSQL** | Structured operational data | `documents`, `document_intelligence`, `document_impact_records`, `global_events`, `macro_impact_records`, `competitive_signal_records`, `trend_windows`, `trend_history`, `trend_projections`, `recommendations`, `recommendation_evidence`, `risk_evaluations`, `orders`, `order_events`, `positions`, `portfolio_snapshots`, `trading_decisions` |
-| **Redis** | Queues, dedup markers, rate limits, circuit breaker state | `stonks:queue:*`, `stonks:dedupe:*`, `stonks:ratelimit:*`, `stonks:trading:circuit_breaker:*`, `stonks:dlq:*` |
-| **MinIO** | Object storage for raw artifacts, normalized text, and analytical Parquet files | Raw artifacts bucket, normalized text bucket, lakehouse bucket (partitioned Parquet) |
+| **PostgreSQL** | Structured operational data | `documents`, `document_intelligence`, `document_impact_records`, `document_company_mentions`, `global_events`, `macro_impact_records`, `exposure_profiles`, `competitive_signal_records`, `competitor_relationships`, `trend_windows`, `trend_history`, `trend_projections`, `recommendations`, `recommendation_evidence`, `risk_evaluations`, `orders`, `order_events`, `positions`, `portfolio_snapshots`, `trading_decisions`, `circuit_breaker_events`, `reserve_pool_ledger`, `risk_tier_history`, `broker_accounts`, `ingestion_runs`, `sources`, `companies`, `company_aliases`, `ai_agents`, `agent_variants`, `agent_performance_log`, `risk_configs`, `trading_reports` |
+| **Redis** | Queues, dedup markers, rate limits, circuit breaker state, pipeline toggle | `stonks:queue:*` (9 queues), `stonks:dedupe:*`, `stonks:dedupe:trading:*`, `stonks:ratelimit:*`, `stonks:trading:circuit_breaker:*`, `stonks:trading:notification_rate:*`, `stonks:order_idempotency:*`, `stonks:lock:*`, `stonks:cache:*`, `stonks:retry:*`, `stonks:rec_dedup:*`, `stonks:pipeline:enabled`, `stonks:dlq:*` |
+| **MinIO** | Object storage for raw artifacts, normalized text, and analytical Parquet files | Raw artifacts bucket, normalized text bucket, parser output bucket, lakehouse bucket (partitioned Parquet: documents, extractions, market bars/quotes, orders, fills, positions, PnL, global events, macro impacts, trend projections, competitive signals, competitor relationships, recommendations) |
 
 ## External Integration Points
 
 | Integration | Service | Protocol | Purpose |
 |-------------|---------|----------|---------|
-| **Polygon.io** | Ingestion (via adapters) | HTTPS REST | News articles, market bars, grouped daily data |
-| **SEC EDGAR** | Ingestion (via FilingsDataAdapter) | HTTPS REST | 10-K, 10-Q filings |
-| **Ollama** | Extractor, Recommendation | HTTP `/api/generate` | LLM inference for document extraction, event classification, thesis rewriting |
-| **Alpaca** | Broker Adapter | HTTPS REST | Paper trading order submission, position sync, account state |
+| **Polygon.io** | Ingestion (via PolygonNewsAdapter, PolygonMarketAdapter) | HTTPS REST | News articles, market bars, grouped daily data, intraday bars |
+| **SEC EDGAR** | Ingestion (via SECEdgarAdapter) | HTTPS REST | 10-K, 10-Q filings |
+| **Macro News** | Ingestion (via MacroNewsAdapter) | HTTPS REST | Geopolitical and economic event articles |
+| **Ollama / vLLM** | Extractor, Recommendation | HTTP `/api/generate` | LLM inference for document extraction (document-extractor agent), event classification (event-classifier agent), thesis rewriting (thesis-rewriter agent). Model and variant selected via `AgentConfigResolver` with 60s TTL cache. |
+| **Alpaca** | Broker Adapter | HTTPS REST | Paper/live trading: order submission, position sync, account state, order status polling |
 | **AWS SNS** | Trading Engine (notifications) | boto3 SDK | SMS alerts for circuit breaker trips, order fills, stop-loss triggers |
 | **Gmail** | Trading Engine (notifications) | SMTP (port 587 STARTTLS) | Email alerts for trading events |
-| **Trino** | Query API, Superset | JDBC / HTTP | SQL queries over lakehouse Parquet files |
+| **Trino** | Query API, Superset | HTTP | SQL queries over lakehouse Parquet files via Hive Metastore |
+
+## Pipeline Toggle
+
+The pipeline can be paused globally via the Redis key `stonks:pipeline:enabled`. When set to `"0"`, all queue workers (ingestion, parser, extractor, aggregation, recommendation, broker adapter, lake publisher) enter a sleep loop and stop processing jobs. The scheduler also skips scheduling cycles when the toggle is off. The toggle can be set via the Query API's pipeline control endpoints.
+
+Setting `PIPELINE_DEFAULT_OFF=true` on the scheduler initializes the toggle to OFF on first boot, useful for staged deployments where you want to verify infrastructure before enabling the pipeline.