diff --git a/.kiro/specs/comprehensive-quality-docs/.config.kiro b/.kiro/specs/comprehensive-quality-docs/.config.kiro
new file mode 100644
index 0000000..c779460
--- /dev/null
+++ b/.kiro/specs/comprehensive-quality-docs/.config.kiro
@@ -0,0 +1 @@
+{"specId": "e433350c-baf0-4f4f-a30e-3724f6654090", "workflowType": "requirements-first", "specType": "feature"}
diff --git a/.kiro/specs/comprehensive-quality-docs/design.md b/.kiro/specs/comprehensive-quality-docs/design.md
new file mode 100644
index 0000000..2742c3e
--- /dev/null
+++ b/.kiro/specs/comprehensive-quality-docs/design.md
@@ -0,0 +1,377 @@
+# Design Document: Comprehensive Quality & Documentation
+
+## Overview
+
+This design covers three pillars for the Stonks Oracle platform:
+
+1. **Test Coverage** — Close unit test gaps in the scheduler and ingestion services, fix pre-existing test failures in the extractor module, and achieve a fully green test suite (Requirements 1–4).
+2. **Docker Deployment** — Extend `docker-compose.yml` to include all 13 application services plus the frontend, enabling full-platform local development without Kubernetes (Requirement 5).
+3. **Documentation** — Produce comprehensive documentation covering per-service features, API references, Helm chart configuration, Docker deployment, three Mermaid architecture diagrams, AI agent building, backup/restore, observability, and README resource links (Requirements 6–16).
+
+### Design Rationale
+
+The platform has mature production code across 13 services but uneven test coverage and documentation. The scheduler and ingestion services lack dedicated unit tests — their logic is only exercised through integration tests. Four extractor-related test files have pre-existing failures that block CI. Documentation exists only as a local dev setup guide, a pipeline overview, and a runbook. This initiative fills those gaps systematically.
+
+The approach prioritizes:
+- **Test isolation**: Mock all external dependencies (PostgreSQL, Redis, MinIO, Ollama) so unit tests run fast and deterministically.
+- **Documentation from source**: Generate API references by inspecting actual FastAPI route definitions, Helm values from `values.yaml`, and metrics from `services/shared/metrics.py`.
+- **Docker parity with Kubernetes**: Mirror the Helm chart's service definitions in Docker Compose so both deployment modes stay in sync.
+
+## Architecture
+
+The work does not change the platform's runtime architecture. It adds:
+
+1. **New test files** in `tests/` for scheduler and ingestion unit tests.
+2. **Fixes** to existing test files and/or production code to resolve failures.
+3. **New service definitions** in `docker-compose.yml` using the existing `docker/Dockerfile` with `SERVICE_CMD` build args.
+4. **New documentation files** in `docs/` organized by topic.
+5. **Updated `README.md`** with a documentation index and Mermaid diagram.
+
+```mermaid
+graph TD
+ subgraph "Test Coverage (Reqs 1-4)"
+ T1[tests/test_scheduler_unit.py]
+ T2[tests/test_ingestion_unit.py]
+ T3[Fix test_extractor_prompts.py]
+ T4[Fix test_extractor_schemas.py]
+ T5[Fix test_ollama_client.py]
+ T6[Fix test_filings_adapter.py]
+ end
+
+ subgraph "Docker (Req 5)"
+ D1[docker-compose.yml
+ 13 app services + frontend]
+ end
+
+ subgraph "Documentation (Reqs 6-16)"
+ DOC1[docs/services.md]
+ DOC2[docs/api-reference.md]
+ DOC3[docs/helm-reference.md]
+ DOC4[docs/docker-deployment.md]
+ DOC5[docs/architecture-kubernetes.md]
+ DOC6[docs/architecture-docker-compose.md]
+ DOC7[docs/architecture-data-pipeline.md]
+ DOC8[docs/ai-agents.md]
+ DOC9[docs/backup-restore.md]
+ DOC10[docs/observability.md]
+ DOC11[README.md update]
+ end
+```
+
+## Components and Interfaces
+
+### 1. Scheduler Unit Tests (Requirement 1)
+
+**Target module**: `services/scheduler/app.py`
+
+**Functions to test in isolation**:
+- `get_cadence_for_source(source_type, config)` — Returns polling interval from config or defaults.
+- `compute_backoff(retry_count)` — Exponential backoff with cap.
+- `is_source_due(...)` — Core scheduling logic: determines if a source needs polling based on last run status, timing, retry state.
+- `build_job_payload(source, aliases, now)` — Constructs the ingestion job dict.
+- `schedule_cycle(pool, rds)` — Full scheduling pass (mocked DB/Redis).
+- `check_rate_limit(rds, source_type, now)` — Rate limiting with per-type and global Polygon limits.
+- `recover_stale_documents(pool, rds)` — Re-enqueue orphaned parsed documents.
+- `retry_failed_extractions(pool, rds)` — Re-enqueue failed extractions.
+
+**Mocking strategy**:
+- `asyncpg.Pool` → `AsyncMock` with `.fetch()`, `.fetchrow()`, `.fetchval()`, `.execute()` returning canned records.
+- `redis.asyncio.Redis` → `AsyncMock` with `.rpush()`, `.set()`, `.get()`, `.incr()`, `.expire()`, `.decr()`, `.delete()` tracking calls.
+- Use `unittest.mock.patch` for module-level imports where needed.
+
+**Test file**: `tests/test_scheduler_unit.py`
+
+### 2. Ingestion Unit Tests (Requirement 2)
+
+**Target module**: `services/ingestion/worker.py`
+
+**Functions to test**:
+- `process_job(job, pool, rds, minio_client, adapters)` — Main job processing with various adapter outcomes.
+- Error handling paths: adapter returns `AdapterResult(error=...)`, retry exhaustion, dead-letter routing.
+- Deduplication: content hash already seen in Redis, cross-source document dedup via `dedupe_items`.
+
+**Mocking strategy**:
+- Adapters → `AsyncMock` returning `AdapterResult` with controlled `error`, `items`, `content_hash`, `raw_payload`.
+- `asyncpg.Pool` → `AsyncMock` for `ingestion_runs` INSERT/UPDATE, `persist_ingestion_items`, `record_retrieval_failure`.
+- `redis.asyncio.Redis` → `AsyncMock` for dedupe checks, queue pushes, DLQ routing.
+- `minio.Minio` → `MagicMock` for `upload_raw_artifact`.
+
+**Test file**: `tests/test_ingestion_unit.py`
+
+### 3. Extractor Test Fixes (Requirement 3)
+
+**Target files**:
+- `tests/test_extractor_prompts.py`
+- `tests/test_extractor_schemas.py`
+- `tests/test_ollama_client.py`
+- `tests/test_filings_adapter.py`
+
+**Approach**: Run each file individually, diagnose failures, and fix either the test setup (mock configuration, fixture data) or the production code. Preserve original test intent and assertions. If production code changes are needed, add regression tests.
+
+### 4. Full Test Suite Green (Requirement 4)
+
+**Verification**: Run `pytest tests/ -x --tb=short -q` and `ruff check services/` after all fixes. All existing `test_pbt_*` files must remain passing. Any production code fix must include a regression test.
+
+### 5. Docker Compose Application Services (Requirement 5)
+
+**Current state**: `docker-compose.yml` defines 7 infrastructure services (postgres, redis, minio, minio-init, ollama, trino, hive-metastore, superset).
+
+**Addition**: 14 new service definitions (13 app services + frontend dashboard):
+
+| Service | Image Build | Command | Port | Depends On |
+|---------|------------|---------|------|------------|
+| scheduler | `docker/Dockerfile.scheduler` | `python -m services.scheduler.app` | — | postgres, redis |
+| symbol-registry | `docker/Dockerfile` | `uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000` | 8001:8000 | postgres |
+| ingestion | `docker/Dockerfile` | `python -m services.ingestion.worker` | — | postgres, redis, minio |
+| parser | `docker/Dockerfile` | `python -m services.parser.worker` | — | postgres, redis |
+| extractor | `docker/Dockerfile` | `python -m services.extractor.main` | — | postgres, redis, ollama |
+| aggregation | `docker/Dockerfile` | `python -m services.aggregation.main` | — | postgres, redis |
+| recommendation | `docker/Dockerfile` | `python -m services.recommendation.main` | — | postgres, redis |
+| trading-engine | `docker/Dockerfile` | `uvicorn services.trading.app:app --host 0.0.0.0 --port 8000` | 8002:8000 | postgres, redis |
+| risk-engine | `docker/Dockerfile` | `uvicorn services.risk.app:app --host 0.0.0.0 --port 8000` | 8003:8000 | postgres |
+| broker-adapter | `docker/Dockerfile` | `python -m services.adapters.broker_service` | — | postgres, redis |
+| lake-publisher | `docker/Dockerfile` | `python -m services.lake_publisher.jobs` | — | postgres, minio |
+| query-api | `docker/Dockerfile` | `uvicorn services.api.app:app --host 0.0.0.0 --port 8000` | 8004:8000 | postgres, redis, minio |
+| dashboard | `frontend/Dockerfile` | nginx (built-in) | 3000:8080 | query-api |
+
+**Common environment block** (shared via `x-app-env` YAML anchor):
+```yaml
+POSTGRES_HOST: postgres
+POSTGRES_PORT: "5432"
+POSTGRES_DB: stonks
+POSTGRES_USER: stonks
+POSTGRES_PASSWORD: stonks_dev
+REDIS_HOST: redis
+REDIS_PORT: "6379"
+MINIO_ENDPOINT: minio:9000
+MINIO_ACCESS_KEY: minioadmin
+MINIO_SECRET_KEY: minioadmin
+OLLAMA_BASE_URL: http://ollama:11434
+```
+
+**`.env` file support**: `MARKET_DATA_API_KEY`, `BROKER_API_KEY`, `BROKER_API_SECRET`, `BROKER_BASE_URL` loaded via `env_file: .env` on services that need them (ingestion, broker-adapter, trading-engine).
+
+**Health checks**: FastAPI services use `curl -f http://localhost:8000/health`; workers use process liveness checks. Infrastructure `depends_on` uses `condition: service_healthy`.
+
+### 6. Documentation Structure (Requirements 6–16)
+
+All documentation files are Markdown in `docs/`. The structure:
+
+```
+docs/
+├── services.md # Req 6: Per-service feature docs
+├── api-reference.md # Req 7: All 4 FastAPI API references
+├── helm-reference.md # Req 8: Helm chart values reference
+├── docker-deployment.md # Req 9: Docker deployment guide
+├── architecture-kubernetes.md # Req 10: K8s Mermaid diagram
+├── architecture-docker-compose.md # Req 11: Docker Compose Mermaid diagram
+├── architecture-data-pipeline.md # Req 12: Data pipeline Mermaid diagram
+├── ai-agents.md # Req 13: AI agent building guide
+├── backup-restore.md # Req 14: Backup and restore guide
+├── observability.md # Req 15: Observability & metrics reference
+├── LOCAL_DEV_SETUP.md # (existing)
+├── llm-to-trade-pipeline.md # (existing)
+└── notes/
+ └── runbook.md # (existing)
+```
+
+#### 6a. Service Feature Documentation (`docs/services.md`) — Req 6
+
+For each of the 13 services, document:
+- **Purpose**: What the service does in the pipeline.
+- **Entry point**: Module path (e.g., `services.scheduler.app`).
+- **Configuration**: Environment variables from `services/shared/config.py` relevant to this service.
+- **Database tables**: Tables read/written by this service.
+- **Redis queues**: Queue names consumed from and published to (from `services/shared/redis_keys.py`).
+- **Queue message schema**: JSON structure of messages.
+- **Signal layers**: For aggregation/recommendation, document the three signal layers (company, macro, competitive), their toggles (`macro_enabled`, `competitive_enabled` in `risk_configs`), and weight configurations.
+- **Trading engine features**: For the trading service, document position sizing, circuit breakers, reserve pool, risk tier auto-adjustment, backtesting, and notification configuration.
+
+Queue topology reference (from `redis_keys.py`):
+| Queue | Producer | Consumer |
+|-------|----------|----------|
+| `stonks:queue:ingestion` | scheduler | ingestion |
+| `stonks:queue:parsing` | ingestion | parser |
+| `stonks:queue:extraction` | parser | extractor |
+| `stonks:queue:macro_classification` | parser, scheduler | extractor |
+| `stonks:queue:aggregation` | extractor | aggregation |
+| `stonks:queue:recommendation` | aggregation | recommendation |
+| `stonks:queue:lake_publish` | various | lake-publisher |
+| `stonks:queue:broker_orders` | trading-engine, trading API | broker-adapter |
+| `stonks:queue:trading_decisions` | recommendation | trading-engine |
+
+#### 6b. API Reference (`docs/api-reference.md`) — Req 7
+
+Document all endpoints from the four FastAPI services by inspecting their route definitions:
+
+**Query API** (`services/api/app.py`): ~40+ endpoints covering companies, documents, trends, recommendations, evidence drill-down, orders, positions, portfolio, global events, macro impacts, competitive signals, trend projections, agents, dead-letter queues, pipeline control, SQL explorer, saved queries, audit trail, DevOps metrics, and Prometheus metrics.
+
+**Symbol Registry API** (`services/symbol_registry/app.py`): Companies CRUD, aliases, watchlists, sources, exposure profiles, competitor relationships, competitor inference.
+
+**Trading API** (`services/trading/app.py`): Health/readiness, engine status, config update, pause/resume, reset, decisions audit, performance metrics/history, backtesting, notifications config/history, override orders, debug state.
+
+**Risk API** (`services/risk/app.py`): Order evaluation (`POST /evaluate`), health, pending approvals, approval review, approval expiration.
+
+For each endpoint: method, path, query parameters (type, default, constraints), request body schema, response schema, error codes (4xx/5xx).
+
+#### 6c. Helm Chart Reference (`docs/helm-reference.md`) — Req 8
+
+Document from `infra/helm/stonks-oracle/values.yaml`:
+- `image` block: registry, pullPolicy, tag
+- `pipelineEnabled`: toggle and effect on worker replicas
+- `services` block: per-service structure (replicas, image, command, tier, port, secrets, resources, probes)
+- `config` block: all ConfigMap environment variables with defaults and descriptions
+- `secrets` block: core, broker, market, gmail, dashboard — injection via `--set` flags
+- `ingress` block: className, clusterIssuer, host mappings
+- Analytics stack: trino, hiveMetastore, superset toggles and resources
+- `networkPolicies.enabled`: default-deny-ingress behavior
+- Value override files: `values-beta.yaml`, `values-paper.yaml` and their deployment stages
+
+#### 6d. Docker Deployment Guide (`docs/docker-deployment.md`) — Req 9
+
+- Complete service inventory with images, ports, volumes, environment variables
+- `.env` file format with all required/optional variables
+- Volume mounts and data persistence (pgdata, miniodata, ollama_models, hive_data, superset_data)
+- Health check configurations
+- Dockerfile build arguments (`SERVICE_CMD`)
+- Operational commands: start, stop, restart, logs, scale, reset (`docker compose down -v`)
+
+#### 6e. Architecture Diagrams (Reqs 10–12)
+
+**Kubernetes diagram** (`docs/architecture-kubernetes.md`):
+- `stonks-oracle` namespace with all 13 services grouped by tier (api, processing, trading, orchestration, analytics, frontend)
+- External cluster services in their namespaces (postgresql-service, redis-service, minio-service, ollama-service)
+- Traefik ingress routes to external domains
+- Network policy boundaries
+- Analytics plane (Trino, Hive Metastore, Superset)
+- Helm-managed secrets (core, broker, market, gmail) with consumer mapping
+- Service tier distinction (API with ingress, pipeline workers, trading)
+
+**Docker Compose diagram** (`docs/architecture-docker-compose.md`):
+- All infrastructure + application containers
+- Host port mappings
+- `depends_on` relationships and health check dependencies
+- Named volumes and mount points
+- `.env` file providing API keys
+- Internal Docker network connectivity
+
+**Data Pipeline diagram** (`docs/architecture-data-pipeline.md`):
+- External sources → ingestion → parsing → extraction → aggregation → recommendation → risk → trading → broker
+- Redis queue topology with queue names
+- Three signal layers as distinct paths merging at aggregation
+- Data stores at each stage (MinIO, PostgreSQL, Redis)
+- Trading engine decision loop
+- Analytical branch (lake publisher → MinIO/Parquet → Trino → Superset/Dashboard)
+- External integrations (Ollama, Alpaca, AWS SNS, Gmail)
+
+#### 6f. AI Agent Guide (`docs/ai-agents.md`) — Req 13
+
+- Three built-in agents: document-extractor, event-classifier, thesis-rewriter
+- Per-agent: purpose, input data, output schema, default model, system prompt structure, user prompt template
+- `ai_agents` table schema and registration (system-seeded vs API-created)
+- `agent_variants` table: create, activate, deactivate variants for A/B testing
+- `AgentConfigResolver` module: TTL cache (60s default), COALESCE-based variant override, fallback behavior
+- Performance logging: `agent_performance_log` table, querying for variant comparison
+- API endpoints: CRUD on `/api/agents`, test endpoint `/api/agents/{id}/test`
+- Step-by-step guide: creating a new variant with different model/prompt and activating it
+
+#### 6g. Backup & Restore Guide (`docs/backup-restore.md`) — Req 14
+
+Scripts in `scripts/`:
+- `backup-db.sh`: PostgreSQL dump, CLI args, storage location, retention (keeps last 7)
+- `restore-db.sh`: PostgreSQL restore, service scale-down/up, data loss implications
+- `backup-redis.sh`: Redis RDB snapshot backup
+- `backup.sh`: Combined backup (DB + Redis), `--upload-minio` option
+- `restore.sh`: Combined restore
+- Full nuke-and-rebuild procedure (connection termination, DB drop, Redis flush, redeploy, re-seed)
+- Recommended backup schedules and automation (cron, Kubernetes CronJobs)
+
+#### 6h. Observability Reference (`docs/observability.md`) — Req 15
+
+- `/metrics` endpoint on query-api, Prometheus scrape configuration
+- All metrics from `services/shared/metrics.py`:
+ - **Ingestion**: `stonks_ingestion_jobs_total`, `stonks_ingestion_items_fetched_total`, `stonks_ingestion_items_new_total`, `stonks_ingestion_items_deduped_total`, `stonks_ingestion_errors_total`, `stonks_ingestion_adapter_duration_seconds`
+ - **Parsing**: `stonks_parse_jobs_total`, `stonks_parse_quality_score`, `stonks_parse_low_quality_total`, `stonks_parse_duration_seconds`
+ - **Extraction**: `stonks_extraction_jobs_total`, `stonks_extraction_attempts_total`, `stonks_extraction_retries_total`, `stonks_extraction_duration_seconds`, `stonks_extraction_confidence`, `stonks_extraction_validation_errors_total`, `stonks_extraction_tokens_total`
+ - **Aggregation**: `stonks_aggregation_windows_total`, `stonks_aggregation_signals_total`, `stonks_aggregation_contradiction_score`, `stonks_aggregation_duration_seconds`
+ - **Recommendation**: `stonks_recommendations_total`, `stonks_recommendations_suppressed_total`, `stonks_recommendation_confidence`
+ - **Lake**: `stonks_lake_facts_published_total`, `stonks_lake_publish_duration_seconds`, `stonks_lake_publish_errors_total`, `stonks_lake_publish_bytes_total`
+ - **Trading**: `stonks_orders_submitted_total`, `stonks_orders_rejected_total`, `stonks_orders_filled_total`, `stonks_orders_duplicates_prevented_total`, `stonks_risk_evaluations_total`, `stonks_risk_check_failures_total`, `stonks_positions_synced_total`
+ - **Alerting**: `stonks_alerts_fired_total`, `stonks_alerts_resolved_total`, `stonks_alert_check_duration_seconds`, `stonks_alert_active`
+ - **DLQ**: `stonks_dlq_items_total`, `stonks_dlq_replayed_total`, `stonks_dlq_depth`
+ - **Active**: `stonks_active_jobs`
+- Alerting module (`services/shared/alerting.py`): 4 alert rules (source_failures, schema_failure_spike, analytical_lag, broker_issues), thresholds, evaluation windows, ConfigMap variables
+- Structured JSON logging format, trace context (trace_id, span_id)
+- Dead-letter queue system: queue names (`stonks:dlq:`), routing, replay tooling
+- Recommended Prometheus/Grafana queries
+
+#### 6i. README Update — Req 16
+
+- Add "Documentation" section with links to all docs
+- Replace ASCII architecture diagram with Mermaid or link to diagram docs
+- Preserve all existing content (license, features, tech stack, project structure, deployment)
+
+## Data Models
+
+No new database tables or schema changes are introduced. This initiative works with existing tables:
+
+**Tables referenced in test coverage work**:
+- `sources`, `companies`, `company_aliases` — scheduler source polling
+- `ingestion_runs` — scheduler run tracking, ingestion job recording
+- `documents`, `document_company_mentions` — ingestion persistence, stale document recovery
+- `document_intelligence`, `document_impact_records` — extractor test fixtures
+- `model_performance_metrics` — extractor schema validation metrics
+
+**Tables documented** (not modified):
+- All tables listed above plus `trend_windows`, `trend_history`, `trend_projections`, `recommendations`, `recommendation_evidence`, `risk_evaluations`, `orders`, `order_events`, `positions`, `portfolio_snapshots`, `trading_decisions`, `circuit_breaker_events`, `reserve_pool_ledger`, `risk_tier_history`, `backtest_runs`, `backtest_trades`, `notifications`, `global_events`, `macro_impact_records`, `exposure_profiles`, `competitor_relationships`, `competitive_signal_records`, `ai_agents`, `agent_variants`, `agent_performance_log`, `audit_events`, `watchlists`, `watchlist_members`, `retention_policies`, `market_snapshots`
+
+## Error Handling
+
+### Test Coverage
+- **Mock failures**: Unit tests must verify that scheduler and ingestion services handle database/Redis connection failures gracefully (no crashes, proper logging).
+- **Adapter errors**: Ingestion unit tests must verify retry logic with exponential backoff and dead-letter queue routing after retry exhaustion.
+- **Test fix approach**: When fixing pre-existing failures, prefer fixing test setup over changing production code. If production code changes are needed, add regression tests to prevent re-introduction.
+
+### Docker Compose
+- **Health check failures**: Application services use `depends_on` with `condition: service_healthy` to wait for infrastructure. Health checks have `interval`, `timeout`, `retries`, and `start_period` configured.
+- **Missing `.env` file**: Services that need API keys (ingestion, broker-adapter, trading-engine) will start but log warnings about missing keys. The platform runs in a degraded mode without external API access.
+- **Build failures**: Each service uses the same base Dockerfile with `SERVICE_CMD` build arg. Build errors are isolated per service.
+
+### Documentation
+- **Stale documentation**: Documentation is generated from source code inspection. If the codebase changes after documentation is written, the docs may drift. The README links section serves as a single index to find and update docs.
+- **Diagram accuracy**: Mermaid diagrams are hand-authored based on current architecture. They should be updated when services are added or removed.
+
+## Testing Strategy
+
+### PBT Applicability Assessment
+
+Property-based testing is **NOT applicable** to this feature. The work consists of:
+1. **Unit tests for existing services** — These are example-based tests with mocked dependencies, not pure functions with universal properties.
+2. **Fixing pre-existing test failures** — Bug fixes to existing tests/code.
+3. **Docker Compose configuration** — Declarative infrastructure configuration.
+4. **Documentation** — Markdown files with no executable logic.
+
+None of these involve new pure functions, parsers, serializers, or business logic where PBT would add value. The existing `test_pbt_*` files (22 files covering trading, aggregation, competitive intelligence, etc.) already provide PBT coverage for the platform's core logic and must remain passing.
+
+### Unit Testing Strategy
+
+**New test files**:
+- `tests/test_scheduler_unit.py` — 8+ test cases covering all scheduler pure functions and the `schedule_cycle` orchestration with mocked dependencies.
+- `tests/test_ingestion_unit.py` — 6+ test cases covering adapter error handling, retry logic, deduplication, and dead-letter queue routing.
+
+**Test fix files** (existing, to be repaired):
+- `tests/test_extractor_prompts.py`
+- `tests/test_extractor_schemas.py`
+- `tests/test_ollama_client.py`
+- `tests/test_filings_adapter.py`
+
+**Test framework**: pytest + pytest-asyncio (already configured in the project).
+
+**Mocking approach**: `unittest.mock.AsyncMock` for async dependencies, `unittest.mock.MagicMock` for sync dependencies, `unittest.mock.patch` for module-level state.
+
+### Verification Criteria
+
+1. `pytest tests/ -x --tb=short -q` → zero failures
+2. `ruff check services/` → zero violations
+3. All 22 existing `test_pbt_*` files pass unchanged
+4. `docker compose config` validates the updated docker-compose.yml
+5. All documentation files render valid Markdown with working internal links
diff --git a/.kiro/specs/comprehensive-quality-docs/requirements.md b/.kiro/specs/comprehensive-quality-docs/requirements.md
new file mode 100644
index 0000000..c63898b
--- /dev/null
+++ b/.kiro/specs/comprehensive-quality-docs/requirements.md
@@ -0,0 +1,236 @@
+# Requirements Document
+
+## Introduction
+
+This initiative covers three pillars for the Stonks Oracle platform: (1) closing unit test coverage gaps across all 13 services, fixing pre-existing test failures, and ensuring every feature has proper automated tests; (2) updating the Docker Compose deployment to include all application services so users can run the full platform without Kubernetes; and (3) producing comprehensive documentation covering every feature, all API endpoints, Helm chart configuration, Docker deployment options, and three Mermaid architecture diagrams (Kubernetes deployment, Docker Compose deployment, and data pipeline), with the README updated to link to all resources.
+
+## Glossary
+
+- **Test_Suite**: The collection of pytest unit tests, property-based tests, and integration tests in the `tests/` directory
+- **Docker_Compose_Stack**: The `docker-compose.yml` file and associated Dockerfiles that define the local development environment
+- **Helm_Chart**: The Kubernetes deployment configuration at `infra/helm/stonks-oracle/` including `values.yaml`, value overrides, and templates
+- **Query_API**: The FastAPI REST service at `services/api/app.py` serving analytics and dashboard queries
+- **Symbol_Registry_API**: The FastAPI REST service at `services/symbol_registry/app.py` managing companies, watchlists, sources, exposure profiles, and competitor relationships
+- **Trading_API**: The FastAPI REST service at `services/trading/app.py` controlling the autonomous trading engine
+- **Risk_API**: The FastAPI REST service at `services/risk/app.py` evaluating order risk and managing approval workflows
+- **Scheduler_Service**: The service at `services/scheduler/` that triggers ingestion cycles on a cadence
+- **Ingestion_Service**: The queue worker at `services/ingestion/` that fetches market data, news, filings, and macro events
+- **Extractor_Service**: The queue worker at `services/extractor/` that performs LLM-based intelligence extraction and event classification
+- **Documentation_Set**: The collection of Markdown files in `docs/` that describe features, APIs, deployment, and architecture
+- **Architecture_Diagram**: A Mermaid-syntax diagram showing services, data stores, external integrations, and data flow. Three diagrams are produced: Kubernetes deployment, Docker Compose deployment, and data pipeline
+- **README**: The root `README.md` file serving as the project entry point
+
+## Requirements
+
+### Requirement 1: Scheduler Service Unit Tests
+
+**User Story:** As a developer, I want the scheduler service to have dedicated unit tests, so that scheduling logic, cadence management, and source polling behavior are verified independently of integration tests.
+
+#### Acceptance Criteria
+
+1. WHEN the Test_Suite is executed for the scheduler module, THE Test_Suite SHALL include unit tests covering job enqueue logic, polling interval calculation, and source due-date evaluation
+2. WHEN a scheduler unit test is run, THE Test_Suite SHALL mock all external dependencies (PostgreSQL, Redis) and test scheduling logic in isolation
+3. THE Test_Suite SHALL verify that the scheduler correctly enqueues ingestion jobs for sources whose polling interval has elapsed
+4. IF a database or Redis connection fails during scheduling, THEN THE Test_Suite SHALL verify that the Scheduler_Service handles the error without crashing
+
+### Requirement 2: Ingestion Service Unit Tests
+
+**User Story:** As a developer, I want the ingestion service to have unit tests for adapter error handling and retry logic, so that data fetching resilience is verified beyond integration tests.
+
+#### Acceptance Criteria
+
+1. WHEN the Test_Suite is executed for the ingestion module, THE Test_Suite SHALL include unit tests covering adapter error handling, retry logic, and deduplication behavior
+2. WHEN an external API returns an error response, THE Test_Suite SHALL verify that the Ingestion_Service retries according to the configured backoff policy
+3. WHEN a duplicate content hash is detected, THE Test_Suite SHALL verify that the Ingestion_Service skips re-processing the document
+4. IF all retry attempts are exhausted, THEN THE Test_Suite SHALL verify that the Ingestion_Service routes the failed job to the dead-letter queue
+
+### Requirement 3: Extractor Test Failure Fixes
+
+**User Story:** As a developer, I want the pre-existing test failures in the extractor module to be resolved, so that the full test suite passes cleanly in CI.
+
+#### Acceptance Criteria
+
+1. WHEN the Test_Suite is executed, THE Test_Suite SHALL pass all tests in `test_extractor_prompts.py` without failures
+2. WHEN the Test_Suite is executed, THE Test_Suite SHALL pass all tests in `test_extractor_schemas.py` without failures
+3. WHEN the Test_Suite is executed, THE Test_Suite SHALL pass all tests in `test_ollama_client.py` without failures
+4. WHEN the Test_Suite is executed, THE Test_Suite SHALL pass all tests in `test_filings_adapter.py` without failures
+5. THE Test_Suite SHALL maintain the original test intent and assertions when fixing failures, modifying only the code under test or test setup as needed
+
+### Requirement 4: Full Test Suite Green Status
+
+**User Story:** As a developer, I want the entire test suite to pass, so that CI builds succeed and regressions are caught immediately.
+
+#### Acceptance Criteria
+
+1. WHEN `pytest tests/ -x --tb=short -q` is executed, THE Test_Suite SHALL report zero failures across all test files
+2. WHEN `ruff check services/` is executed, THE Test_Suite SHALL report zero lint violations
+3. THE Test_Suite SHALL maintain all existing property-based tests (files prefixed `test_pbt_*`) in a passing state
+4. IF a test fix requires modifying production code, THEN THE Test_Suite SHALL include a regression test that validates the fix
+
+### Requirement 5: Docker Compose Application Services
+
+**User Story:** As a developer using Docker instead of Kubernetes, I want docker-compose.yml to include all 13 application services and the frontend, so that I can run the full platform locally with a single `docker compose up`.
+
+#### Acceptance Criteria
+
+1. THE Docker_Compose_Stack SHALL define service containers for all 13 application services: scheduler, symbol-registry, ingestion, parser, extractor, aggregation, recommendation, trading-engine, risk-engine, broker-adapter, lake-publisher, query-api, and dashboard
+2. THE Docker_Compose_Stack SHALL define a frontend container serving the React dashboard via nginx on port 8080
+3. WHEN `docker compose up` is executed, THE Docker_Compose_Stack SHALL start all infrastructure services (PostgreSQL, Redis, MinIO, Ollama, Trino, Hive Metastore, Superset) before application services using dependency ordering
+4. WHEN an application service container starts, THE Docker_Compose_Stack SHALL provide health checks that verify the service is ready to accept requests
+5. THE Docker_Compose_Stack SHALL configure environment variables for each service matching the defaults documented in `docs/LOCAL_DEV_SETUP.md`, with infrastructure hostnames pointing to Docker Compose service names
+6. THE Docker_Compose_Stack SHALL allow users to provide API keys (MARKET_DATA_API_KEY, BROKER_API_KEY, BROKER_API_SECRET) via a `.env` file without modifying docker-compose.yml
+7. IF an infrastructure dependency (PostgreSQL, Redis) is not yet healthy, THEN THE Docker_Compose_Stack SHALL delay application service startup using `depends_on` with `condition: service_healthy`
+
+### Requirement 6: Service Feature Documentation
+
+**User Story:** As a user or contributor, I want every service documented with its purpose, configuration, queue interactions, and database tables, so that I can understand how each part of the platform works.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include a dedicated document for each of the 13 services describing its purpose, inputs, outputs, configuration environment variables, and database tables used
+2. WHEN a service consumes from or publishes to a Redis queue, THE Documentation_Set SHALL document the queue name, message schema, and processing behavior
+3. WHEN a service exposes HTTP endpoints, THE Documentation_Set SHALL reference the API documentation for that service
+4. THE Documentation_Set SHALL describe the three signal layers (company, macro, competitive) with their data flow, toggle mechanisms, and weight configurations
+5. THE Documentation_Set SHALL document the trading engine features including position sizing, circuit breakers, reserve pool management, risk tier auto-adjustment, backtesting, and notification configuration
+
+### Requirement 7: API Reference Documentation
+
+**User Story:** As a developer integrating with Stonks Oracle, I want a complete API reference for all four FastAPI services, so that I know every endpoint, its parameters, request/response schemas, and error codes.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include an API reference document covering all endpoints of the Query_API, including path, method, query parameters, response schema, and error codes
+2. THE Documentation_Set SHALL include an API reference document covering all endpoints of the Symbol_Registry_API, including CRUD operations for companies, aliases, watchlists, sources, exposure profiles, and competitor relationships
+3. THE Documentation_Set SHALL include an API reference document covering all endpoints of the Trading_API, including engine control, decision audit, performance metrics, backtesting, notifications, and manual override orders
+4. THE Documentation_Set SHALL include an API reference document covering all endpoints of the Risk_API, including order evaluation, approval workflow, and approval expiration
+5. WHEN an endpoint accepts query parameters or a request body, THE Documentation_Set SHALL document each parameter with its type, default value, and constraints
+6. WHEN an endpoint returns an error, THE Documentation_Set SHALL document the HTTP status code and error response format
+
+### Requirement 8: Helm Chart Configuration Reference
+
+**User Story:** As an operator deploying Stonks Oracle on Kubernetes, I want a complete reference for all Helm chart values, so that I can configure services, resources, secrets, ingress, network policies, and analytics components.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include a Helm configuration reference documenting every key in `values.yaml` with its type, default value, and description
+2. THE Documentation_Set SHALL document the `services` block structure including replicas, image, command, tier, port, secrets, resources, and probes for each service
+3. THE Documentation_Set SHALL document the `config` block with all ConfigMap environment variables, their defaults, and what they control
+4. THE Documentation_Set SHALL document the `secrets` block structure (core, broker, market, gmail, dashboard) and how secrets are injected via `--set` flags during deployment
+5. THE Documentation_Set SHALL document the `ingress` block including className, clusterIssuer, and host mappings
+6. THE Documentation_Set SHALL document the analytics stack toggles (trino.enabled, hiveMetastore.enabled, superset.enabled) and their resource configurations
+7. THE Documentation_Set SHALL document the `pipelineEnabled` toggle and its effect on worker service replicas
+8. THE Documentation_Set SHALL document the `networkPolicies.enabled` toggle and the default-deny-ingress behavior
+9. THE Documentation_Set SHALL document the value override files (`values-beta.yaml`, `values-paper.yaml`) and their intended deployment stages
+
+### Requirement 9: Docker Deployment Guide
+
+**User Story:** As a developer deploying with Docker Compose, I want a guide explaining all Docker deployment options, environment variables, volume mounts, and operational commands, so that I can run and manage the platform without Kubernetes.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include a Docker deployment guide documenting every service defined in docker-compose.yml with its image, ports, volumes, and environment variables
+2. THE Documentation_Set SHALL document the `.env` file format with all required and optional environment variables, their defaults, and descriptions
+3. THE Documentation_Set SHALL document volume mounts and data persistence behavior, including how to reset data with `docker compose down -v`
+4. THE Documentation_Set SHALL document health check configurations and how to verify all services are running
+5. THE Documentation_Set SHALL document the Dockerfile build arguments (SERVICE_CMD) and how to build custom service images
+6. THE Documentation_Set SHALL document operational commands for starting, stopping, restarting individual services, viewing logs, and scaling replicas
+
+### Requirement 10: Kubernetes Architecture Diagram
+
+**User Story:** As an operator deploying on Kubernetes, I want a Mermaid diagram showing how Stonks Oracle runs in a K8s cluster, so that I can understand the deployment topology, networking, and infrastructure dependencies.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include a Mermaid diagram showing all 13 application services deployed as Kubernetes Deployments within the `stonks-oracle` namespace
+2. THE diagram SHALL show external cluster services (PostgreSQL, Redis, MinIO, Ollama) in their respective namespaces with cross-namespace service references
+3. THE diagram SHALL show Traefik ingress routes mapping external domains to internal services (stonks.celestium.life → dashboard, stonks-api.celestium.life → query-api, etc.)
+4. THE diagram SHALL show network policy boundaries indicating which services can communicate with each other
+5. THE diagram SHALL show the analytics plane (Trino, Hive Metastore, Superset) deployed within the stonks-oracle namespace and their connections to MinIO
+6. THE diagram SHALL show Helm-managed secrets (core, broker, market, gmail) and which services consume them
+7. THE diagram SHALL distinguish between API-tier services (with ingress), pipeline-tier workers (queue-driven), and trading-tier services
+
+### Requirement 11: Docker Compose Architecture Diagram
+
+**User Story:** As a developer running the platform locally with Docker Compose, I want a Mermaid diagram showing how all containers are wired together, so that I can understand port mappings, volume mounts, and service dependencies.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include a Mermaid diagram showing all infrastructure containers (PostgreSQL, Redis, MinIO, Ollama, Trino, Hive Metastore, Superset) and all 13 application service containers as defined in docker-compose.yml
+2. THE diagram SHALL show host port mappings for externally accessible services (PostgreSQL:5432, Redis:6379, MinIO:9000/9001, Ollama:11434, Trino:8080, Superset:8088, Dashboard:8080, Query API:8000)
+3. THE diagram SHALL show Docker Compose `depends_on` relationships and health check dependencies between infrastructure and application services
+4. THE diagram SHALL show named volumes (pgdata, miniodata, ollama_models, hive_data, superset_data) and which containers mount them
+5. THE diagram SHALL show the `.env` file providing API keys (MARKET_DATA_API_KEY, BROKER_API_KEY, BROKER_API_SECRET) to relevant service containers
+6. THE diagram SHALL show internal Docker network connectivity between containers using Docker Compose service names as hostnames
+
+### Requirement 12: Data Pipeline Architecture Diagram
+
+**User Story:** As a user or contributor, I want a Mermaid diagram showing the end-to-end data pipeline from external data sources through signal processing to trade execution, so that I can understand how data flows through the system.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include a Mermaid diagram showing the complete data pipeline from external sources (Polygon.io, news APIs, SEC filings, macro news sources) through ingestion, parsing, extraction, aggregation, recommendation, risk evaluation, and trade execution
+2. THE diagram SHALL show the Redis queue topology connecting pipeline stages (ingestion → parsing → extraction → aggregation → recommendation → broker) with queue names
+3. THE diagram SHALL show the three signal layers (company, macro, competitive) as distinct processing paths that merge in the aggregation stage
+4. THE diagram SHALL show data stores at each stage: MinIO for raw artifacts, PostgreSQL for structured data, Redis for queues and caching
+5. THE diagram SHALL show the trading engine decision loop: recommendation polling → position sizing → risk evaluation → order execution → broker submission → fill tracking
+6. THE diagram SHALL show the analytical branch: lake publisher writing Parquet fact tables to MinIO, queryable via Trino, visualized in Superset and the React dashboard
+7. THE diagram SHALL show external integrations at their connection points: Ollama for LLM extraction, Alpaca for trade execution, AWS SNS and Gmail for notifications
+
+### Requirement 13: AI Agent Building Guide
+
+**User Story:** As a user or contributor, I want a guide explaining how each of the three AI agents works — document extractor, event classifier, and thesis rewriter — including how to configure them, create variants, tune prompts, and monitor performance, so that I can customize and extend the AI capabilities of the platform.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include an AI agent guide documenting the three built-in agents: `document-extractor` (structured intelligence extraction from news/filings), `event-classifier` (macro/geopolitical event classification), and `thesis-rewriter` (LLM-enhanced recommendation thesis generation)
+2. FOR each agent, THE Documentation_Set SHALL document its purpose, input data, output schema, default model, system prompt structure, and user prompt template
+3. THE Documentation_Set SHALL document the `ai_agents` database table schema and how agents are registered (system-seeded vs user-created via the API)
+4. THE Documentation_Set SHALL document the `agent_variants` table and how to create, activate, and deactivate variants for A/B testing different models or prompts
+5. THE Documentation_Set SHALL document the `AgentConfigResolver` module including the TTL cache (60-second default), COALESCE-based variant override logic, and fallback behavior when no DB config exists
+6. THE Documentation_Set SHALL document the agent performance logging system and how to query `agent_performance_log` to compare variant effectiveness
+7. THE Documentation_Set SHALL document the API endpoints for managing agents (CRUD on `/api/agents`) and testing agent configurations (`/api/agents/{id}/test`)
+8. THE Documentation_Set SHALL include a step-by-step guide for creating a new agent variant with a different model or prompt and activating it for live traffic
+
+### Requirement 14: Backup and Restore Guide
+
+**User Story:** As an operator, I want a guide documenting all backup and restore scripts, their options, storage locations, and retention policies, so that I can protect data and recover from failures.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include a backup and restore guide documenting every script in `scripts/` related to backup and restore: `backup-db.sh`, `restore-db.sh`, `backup-redis.sh`, `backup.sh`, and `restore.sh`
+2. FOR each backup script, THE Documentation_Set SHALL document its CLI arguments, what data it captures, where backups are stored, and retention/pruning behavior (e.g., keeps last 7)
+3. FOR each restore script, THE Documentation_Set SHALL document its CLI arguments, what it restores, the service scale-down/scale-up procedure it performs, and any data loss implications
+4. THE Documentation_Set SHALL document the MinIO upload option (`--upload-minio`) for off-host backup storage
+5. THE Documentation_Set SHALL document the full database nuke and rebuild procedure including connection termination, database drop, Redis flush, redeploy, and re-seed steps
+6. THE Documentation_Set SHALL document recommended backup schedules and how to automate backups via cron or Kubernetes CronJobs
+
+### Requirement 15: Observability and Prometheus Metrics Reference
+
+**User Story:** As an operator, I want a reference documenting all Prometheus metrics exposed by the platform, the alerting rules, and how to monitor pipeline health, so that I can set up dashboards and respond to incidents.
+
+#### Acceptance Criteria
+
+1. THE Documentation_Set SHALL include an observability reference documenting the `/metrics` endpoint on the query API and how to configure Prometheus to scrape it
+2. THE Documentation_Set SHALL document all Prometheus counters, gauges, and histograms emitted by each service, including metric name, labels, and what they measure (e.g., `EXTRACTION_ATTEMPTS`, `EXTRACTION_DURATION`, `AGGREGATION_WINDOWS_COMPUTED`, `AGGREGATION_SIGNALS_PROCESSED`, `RECOMMENDATION_GENERATED`, `RECOMMENDATION_CONFIDENCE`, alerting counters)
+3. THE Documentation_Set SHALL document the alerting module (`services/shared/alerting.py`) including all alert rules, their thresholds, evaluation windows, and the ConfigMap environment variables that control them (`ALERT_SOURCE_FAILURE_THRESHOLD`, `ALERT_SCHEMA_FAILURE_RATE_THRESHOLD`, `ALERT_LAKE_LAG_THRESHOLD_MINUTES`, `ALERT_BROKER_ERROR_THRESHOLD`, etc.)
+4. THE Documentation_Set SHALL document the structured JSON logging format, trace context propagation (trace_id, span_id), and how to query logs for debugging pipeline issues
+5. THE Documentation_Set SHALL document the dead-letter queue system including queue names, how failed jobs are routed there, and how to replay them using the dead-letter tooling
+6. THE Documentation_Set SHALL document recommended Prometheus/Grafana dashboard configurations or queries for monitoring ingestion throughput, extraction latency, aggregation volume, recommendation generation rate, and trading engine activity
+
+### Requirement 16: README Resource Links
+
+**User Story:** As a user landing on the repository, I want the README to link to all documentation resources, so that I can navigate to any guide, reference, or diagram from a single entry point.
+
+#### Acceptance Criteria
+
+1. WHEN the README is updated, THE README SHALL include a documentation section with links to every document in the Documentation_Set
+2. THE README SHALL link to the API reference documents for all four FastAPI services
+3. THE README SHALL link to the Helm chart configuration reference
+4. THE README SHALL link to the Docker deployment guide
+5. THE README SHALL link to all three architecture diagram documents (Kubernetes, Docker Compose, and Data Pipeline)
+6. THE README SHALL link to the per-service feature documentation
+7. THE README SHALL link to the AI agent building guide
+8. THE README SHALL link to the backup and restore guide
+9. THE README SHALL link to the observability and Prometheus metrics reference
+10. THE README SHALL replace the existing ASCII architecture diagram with the Mermaid architecture diagram or link to it
+11. THE README SHALL preserve all existing content (license, features, tech stack, project structure, deployment instructions) while adding the new documentation links
diff --git a/.kiro/specs/comprehensive-quality-docs/tasks.md b/.kiro/specs/comprehensive-quality-docs/tasks.md
new file mode 100644
index 0000000..61773fa
--- /dev/null
+++ b/.kiro/specs/comprehensive-quality-docs/tasks.md
@@ -0,0 +1,223 @@
+# Implementation Plan: Comprehensive Quality & Documentation
+
+## Overview
+
+This plan implements three pillars for the Stonks Oracle platform: (1) unit test coverage for the scheduler and ingestion services plus fixing pre-existing test failures, (2) extending docker-compose.yml with all 13 application services and the frontend, and (3) producing comprehensive documentation covering services, APIs, Helm configuration, Docker deployment, architecture diagrams, AI agents, backup/restore, observability, and README resource links. Tasks are ordered so tests come first (catch regressions early), then Docker Compose (infrastructure), then documentation (references verified code).
+
+## Tasks
+
+- [x] 1. Write scheduler service unit tests
+ - [x] 1.1 Create `tests/test_scheduler_unit.py` with unit tests for scheduler pure functions and orchestration
+ - Import scheduler functions from `services/scheduler/app.py`
+ - Mock `asyncpg.Pool` (`.fetch()`, `.fetchrow()`, `.fetchval()`, `.execute()`) and `redis.asyncio.Redis` (`.rpush()`, `.set()`, `.get()`, `.incr()`, `.expire()`, `.decr()`, `.delete()`)
+ - Write 8+ test cases covering: `get_cadence_for_source`, `compute_backoff`, `is_source_due`, `build_job_payload`, `schedule_cycle` (mocked DB/Redis), `check_rate_limit`, `recover_stale_documents`, `retry_failed_extractions`
+ - Verify error handling: DB/Redis connection failures handled without crashing
+ - Use `pytest-asyncio` for async test functions, `unittest.mock.AsyncMock` and `unittest.mock.patch`
+ - _Requirements: 1.1, 1.2, 1.3, 1.4_
+
+ - [x] 1.2 Write additional edge-case unit tests for scheduler
+ - Test boundary conditions: zero polling interval, max retry count, empty source list
+ - Test rate limiting edge cases: global Polygon limit, per-type limits
+ - _Requirements: 1.3, 1.4_
+
+- [x] 2. Write ingestion service unit tests
+ - [x] 2.1 Create `tests/test_ingestion_unit.py` with unit tests for ingestion worker
+ - Import ingestion functions from `services/ingestion/worker.py`
+ - Mock adapters as `AsyncMock` returning `AdapterResult` with controlled `error`, `items`, `content_hash`, `raw_payload`
+ - Mock `asyncpg.Pool` for `ingestion_runs` INSERT/UPDATE, `persist_ingestion_items`, `record_retrieval_failure`
+ - Mock `redis.asyncio.Redis` for dedupe checks, queue pushes, DLQ routing
+ - Mock `minio.Minio` for `upload_raw_artifact`
+ - Write 6+ test cases covering: successful job processing, adapter error with retry, retry exhaustion → dead-letter queue, content hash deduplication skip, cross-source dedup via `dedupe_items`, error handling paths
+ - _Requirements: 2.1, 2.2, 2.3, 2.4_
+
+ - [x] 2.2 Write additional edge-case unit tests for ingestion
+ - Test empty adapter response, partial failures, multiple items in single job
+ - _Requirements: 2.1, 2.4_
+
+- [x] 3. Checkpoint — Verify new unit tests pass
+ - Run `pytest tests/test_scheduler_unit.py tests/test_ingestion_unit.py -x --tb=short -q`
+ - Ensure all tests pass, ask the user if questions arise.
+
+- [x] 4. Fix pre-existing test failures
+ - [x] 4.1 Fix `tests/test_extractor_prompts.py`
+ - Run the file individually to diagnose failures
+ - Fix test setup (mock configuration, fixture data) or production code as needed
+ - Preserve original test intent and assertions
+ - If production code changes are needed, add regression tests
+ - _Requirements: 3.1, 3.5_
+
+ - [x] 4.2 Fix `tests/test_extractor_schemas.py`
+ - Run the file individually to diagnose failures
+ - Fix test setup or production code as needed
+ - Preserve original test intent and assertions
+ - _Requirements: 3.2, 3.5_
+
+ - [x] 4.3 Fix `tests/test_ollama_client.py`
+ - Run the file individually to diagnose failures
+ - Fix test setup or production code as needed
+ - Preserve original test intent and assertions
+ - _Requirements: 3.3, 3.5_
+
+ - [x] 4.4 Fix `tests/test_filings_adapter.py`
+ - Run the file individually to diagnose failures
+ - Fix test setup or production code as needed
+ - Preserve original test intent and assertions
+ - _Requirements: 3.4, 3.5_
+
+- [x] 5. Checkpoint — Full test suite green
+ - Run `pytest tests/ -x --tb=short -q` and verify zero failures
+ - Run `ruff check services/` and verify zero violations
+ - Verify all `test_pbt_*` files pass unchanged
+ - If any production code was modified, confirm regression tests exist
+ - Ensure all tests pass, ask the user if questions arise.
+ - _Requirements: 4.1, 4.2, 4.3, 4.4_
+
+- [x] 6. Add application services to docker-compose.yml
+ - [x] 6.1 Add shared environment anchor and all 14 service definitions to `docker-compose.yml`
+ - Define `x-app-env` YAML anchor with common environment variables (POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, REDIS_HOST, REDIS_PORT, MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, OLLAMA_BASE_URL)
+ - Add 13 application service definitions: scheduler (using `docker/Dockerfile.scheduler`), symbol-registry, ingestion, parser, extractor, aggregation, recommendation, trading-engine, risk-engine, broker-adapter, lake-publisher, query-api — each using `docker/Dockerfile` with appropriate `SERVICE_CMD` build arg
+ - Add dashboard service using `frontend/Dockerfile` on port 3000:8080
+ - Configure `depends_on` with `condition: service_healthy` for infrastructure dependencies
+ - Add health checks: FastAPI services use `curl -f http://localhost:8000/health`, workers use process liveness
+ - Configure `env_file: .env` on services needing API keys (ingestion, broker-adapter, trading-engine)
+ - Map host ports: symbol-registry:8001, trading-engine:8002, risk-engine:8003, query-api:8004, dashboard:3000
+ - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7_
+
+ - [x] 6.2 Validate docker-compose.yml configuration
+ - Run `docker compose config` to verify the updated file parses correctly
+ - _Requirements: 5.1_
+
+- [x] 7. Checkpoint — Tests and Docker Compose validated
+ - Run `pytest tests/ -x --tb=short -q` to confirm no regressions
+ - Run `docker compose config` to confirm valid YAML
+ - Ensure all tests pass, ask the user if questions arise.
+
+- [x] 8. Write per-service feature documentation
+ - [x] 8.1 Create `docs/services.md` documenting all 13 services
+ - For each service: purpose, entry point module path, configuration environment variables, database tables read/written, Redis queues consumed/published with message schemas
+ - Include queue topology table (queue name → producer → consumer)
+ - Document the three signal layers (company, macro, competitive) with data flow, toggles, and weight configurations
+ - Document trading engine features: position sizing, circuit breakers, reserve pool, risk tier auto-adjustment, backtesting, notifications
+ - Cross-reference API documentation for services with HTTP endpoints
+ - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
+
+- [x] 9. Write API reference documentation
+ - [x] 9.1 Create `docs/api-reference.md` covering all four FastAPI services
+ - Document all Query API endpoints (~40+): path, method, query parameters (type, default, constraints), request body schema, response schema, error codes
+ - Document all Symbol Registry API endpoints: companies CRUD, aliases, watchlists, sources, exposure profiles, competitor relationships, competitor inference
+ - Document all Trading API endpoints: health/readiness, engine status, config update, pause/resume, reset, decisions audit, performance metrics/history, backtesting, notifications config/history, override orders, debug state
+ - Document all Risk API endpoints: order evaluation (POST /evaluate), health, pending approvals, approval review, approval expiration
+ - Inspect actual route definitions in `services/api/app.py`, `services/symbol_registry/app.py`, `services/trading/app.py`, `services/risk/app.py`
+ - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_
+
+- [x] 10. Write Helm chart configuration reference
+ - [x] 10.1 Create `docs/helm-reference.md` documenting all Helm values
+ - Document `image` block: registry, pullPolicy, tag
+ - Document `pipelineEnabled` toggle and effect on worker replicas
+ - Document `services` block: per-service structure (replicas, image, command, tier, port, secrets, resources, probes)
+ - Document `config` block: all ConfigMap environment variables with defaults and descriptions
+ - Document `secrets` block: core, broker, market, gmail, dashboard — injection via `--set` flags
+ - Document `ingress` block: className, clusterIssuer, host mappings
+ - Document analytics stack toggles: trino.enabled, hiveMetastore.enabled, superset.enabled with resources
+ - Document `networkPolicies.enabled` and default-deny-ingress behavior
+ - Document value override files: `values-beta.yaml`, `values-paper.yaml` and deployment stages
+ - _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 8.8, 8.9_
+
+- [x] 11. Write Docker deployment guide
+ - [x] 11.1 Create `docs/docker-deployment.md` with complete Docker deployment guide
+ - Document every service with image, ports, volumes, environment variables
+ - Document `.env` file format with all required/optional variables, defaults, descriptions
+ - Document volume mounts and data persistence (pgdata, miniodata, ollama_models, hive_data, superset_data), reset with `docker compose down -v`
+ - Document health check configurations and verification commands
+ - Document Dockerfile build arguments (`SERVICE_CMD`) and custom image builds
+ - Document operational commands: start, stop, restart, logs, scale, reset
+ - _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6_
+
+- [x] 12. Checkpoint — Documentation progress check
+ - Verify `docs/services.md`, `docs/api-reference.md`, `docs/helm-reference.md`, `docs/docker-deployment.md` exist and render valid Markdown
+ - Ensure all tests pass, ask the user if questions arise.
+
+- [x] 13. Write architecture diagrams
+ - [x] 13.1 Create `docs/architecture-kubernetes.md` with Kubernetes deployment Mermaid diagram
+ - Show all 13 services in `stonks-oracle` namespace grouped by tier (api, processing, trading, orchestration, analytics, frontend)
+ - Show external cluster services (PostgreSQL, Redis, MinIO, Ollama) in their namespaces
+ - Show Traefik ingress routes to external domains
+ - Show network policy boundaries
+ - Show analytics plane (Trino, Hive Metastore, Superset) and MinIO connections
+ - Show Helm-managed secrets (core, broker, market, gmail) with consumer mapping
+ - Distinguish API-tier (with ingress), pipeline-tier (queue-driven), and trading-tier services
+ - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7_
+
+ - [x] 13.2 Create `docs/architecture-docker-compose.md` with Docker Compose Mermaid diagram
+ - Show all infrastructure + application containers
+ - Show host port mappings for externally accessible services
+ - Show `depends_on` relationships and health check dependencies
+ - Show named volumes and mount points
+ - Show `.env` file providing API keys to relevant containers
+ - Show internal Docker network connectivity
+ - _Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6_
+
+ - [x] 13.3 Create `docs/architecture-data-pipeline.md` with data pipeline Mermaid diagram
+ - Show complete pipeline: external sources → ingestion → parsing → extraction → aggregation → recommendation → risk → trading → broker
+ - Show Redis queue topology with queue names
+ - Show three signal layers as distinct paths merging at aggregation
+ - Show data stores at each stage (MinIO, PostgreSQL, Redis)
+ - Show trading engine decision loop
+ - Show analytical branch: lake publisher → MinIO/Parquet → Trino → Superset/Dashboard
+ - Show external integrations: Ollama, Alpaca, AWS SNS, Gmail
+ - _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 12.7_
+
+- [x] 14. Write AI agent building guide
+ - [x] 14.1 Create `docs/ai-agents.md` with AI agent guide
+ - Document three built-in agents: document-extractor, event-classifier, thesis-rewriter — purpose, input data, output schema, default model, system prompt structure, user prompt template
+ - Document `ai_agents` table schema and registration (system-seeded vs API-created)
+ - Document `agent_variants` table: create, activate, deactivate variants for A/B testing
+ - Document `AgentConfigResolver` module: TTL cache (60s), COALESCE-based variant override, fallback behavior
+ - Document performance logging: `agent_performance_log` table, querying for variant comparison
+ - Document API endpoints: CRUD on `/api/agents`, test endpoint `/api/agents/{id}/test`
+ - Include step-by-step guide: creating a new variant with different model/prompt and activating it
+ - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7, 13.8_
+
+- [x] 15. Write backup and restore guide
+ - [x] 15.1 Create `docs/backup-restore.md` with backup and restore guide
+ - Document all scripts in `scripts/`: `backup-db.sh`, `restore-db.sh`, `backup-redis.sh`, `backup.sh`, `restore.sh`
+ - For each backup script: CLI arguments, data captured, storage location, retention/pruning (keeps last 7)
+ - For each restore script: CLI arguments, what it restores, service scale-down/up procedure, data loss implications
+ - Document MinIO upload option (`--upload-minio`) for off-host storage
+ - Document full nuke-and-rebuild procedure: connection termination, DB drop, Redis flush, redeploy, re-seed
+ - Document recommended backup schedules and automation (cron, Kubernetes CronJobs)
+ - _Requirements: 14.1, 14.2, 14.3, 14.4, 14.5, 14.6_
+
+- [x] 16. Write observability and metrics reference
+ - [x] 16.1 Create `docs/observability.md` with observability reference
+ - Document `/metrics` endpoint on query-api and Prometheus scrape configuration
+ - Document all Prometheus counters, gauges, histograms from `services/shared/metrics.py` — ingestion, parsing, extraction, aggregation, recommendation, lake, trading, alerting, DLQ, active jobs metrics with names, labels, descriptions
+ - Document alerting module (`services/shared/alerting.py`): 4 alert rules, thresholds, evaluation windows, ConfigMap variables
+ - Document structured JSON logging format, trace context (trace_id, span_id), log querying
+ - Document dead-letter queue system: queue names (`stonks:dlq:`), routing, replay tooling
+ - Document recommended Prometheus/Grafana queries for monitoring
+ - _Requirements: 15.1, 15.2, 15.3, 15.4, 15.5, 15.6_
+
+- [x] 17. Update README with documentation links
+ - [x] 17.1 Update `README.md` with documentation section and resource links
+ - Add "Documentation" section with links to all docs: services.md, api-reference.md, helm-reference.md, docker-deployment.md, architecture-kubernetes.md, architecture-docker-compose.md, architecture-data-pipeline.md, ai-agents.md, backup-restore.md, observability.md
+ - Replace ASCII architecture diagram with Mermaid diagram or link to architecture diagram docs
+ - Preserve all existing content: license, features, tech stack, project structure, deployment instructions
+ - _Requirements: 16.1, 16.2, 16.3, 16.4, 16.5, 16.6, 16.7, 16.8, 16.9, 16.10, 16.11_
+
+- [x] 18. Final checkpoint — Full verification
+ - Run `pytest tests/ -x --tb=short -q` — zero failures
+ - Run `ruff check services/` — zero violations
+ - Run `docker compose config` — validates successfully
+ - Verify all `test_pbt_*` files pass unchanged
+ - Verify all documentation files exist in `docs/` and render valid Markdown
+ - Ensure all tests pass, ask the user if questions arise.
+
+## Notes
+
+- Tasks marked with `*` are optional and can be skipped for faster MVP
+- Each task references specific requirements for traceability
+- Checkpoints ensure incremental validation
+- No property-based tests are included — the design assessment confirmed PBT is not applicable to this feature
+- Existing `test_pbt_*` files (22 files) must remain passing throughout
+- The implementation language is Python (with Markdown for documentation), matching the existing codebase
diff --git a/.kiro/specs/intelligence-pipeline-deep-dive/.config.kiro b/.kiro/specs/intelligence-pipeline-deep-dive/.config.kiro
new file mode 100644
index 0000000..bc874fb
--- /dev/null
+++ b/.kiro/specs/intelligence-pipeline-deep-dive/.config.kiro
@@ -0,0 +1 @@
+{"specId": "d2fe9091-6423-482c-a4ce-3cd72e62eb23", "workflowType": "requirements-first", "specType": "feature"}
diff --git a/.kiro/specs/intelligence-pipeline-deep-dive/design.md b/.kiro/specs/intelligence-pipeline-deep-dive/design.md
new file mode 100644
index 0000000..ebf7d9b
--- /dev/null
+++ b/.kiro/specs/intelligence-pipeline-deep-dive/design.md
@@ -0,0 +1,153 @@
+# Design Document: Intelligence Pipeline Deep Dive
+
+## Overview
+
+This design specifies the structure, content, and creation process for a 6-page narrative deep-dive document covering the full intelligence-to-decision pipeline in Stonks Oracle. The deliverable consists of Markdown narrative pages, an index file, and standalone Mermaid diagram files — all stored under `docs/intelligence-pipeline-deep-dive/`.
+
+The document targets technical readers who want to understand how raw data enters the system, gets processed by AI agents, produces structured signals, accumulates into trend summaries, and ultimately drives autonomous trading decisions. Unlike the existing reference docs (`docs/services.md`, `docs/architecture-data-pipeline.md`), this deliverable is narrative and explanatory — it tells the story of data flowing through the platform end-to-end.
+
+**Key design decision**: This is a documentation-only deliverable. No application code, database schemas, or infrastructure changes are involved. The output is purely Markdown files and Mermaid diagram files.
+
+### Existing Documentation Landscape
+
+The codebase already has several reference documents that this deep-dive complements:
+
+| Document | Purpose | Style |
+|----------|---------|-------|
+| `docs/architecture-data-pipeline.md` | Queue topology, data store summary, Mermaid flow diagrams | Reference diagrams + tables |
+| `docs/llm-to-trade-pipeline.md` | End-to-end data flow from model output to trade | Narrative + tables + code blocks |
+| `docs/services.md` | Per-service configuration, tables, queues, behaviors | Reference manual |
+| `docs/ai-agents.md` | AI agent configuration, variants, A/B testing, API | Guide + reference |
+
+The deep-dive document will reference these existing docs for readers who want deeper detail, while providing a cohesive narrative that connects all pipeline stages into a single story.
+
+## Architecture
+
+### File Organization
+
+```
+docs/intelligence-pipeline-deep-dive/
+├── index.md
+├── 01-data-ingestion-and-preparation.md
+├── 02-ai-agent-processing-and-extraction.md
+├── 03-signal-scoring-and-weighted-signals.md
+├── 04-trend-aggregation-and-accumulating-signals.md
+├── 05-recommendation-generation.md
+├── 06-trading-decisions-and-execution.md
+└── diagrams/
+ ├── ingestion-to-extraction-flow.md
+ ├── three-layer-signal-merging.md
+ ├── recommendation-generation-flow.md
+ ├── trading-engine-decision-loop.md
+ ├── weighted-signal-computation.md
+ └── trend-accumulation-escalation.md
+```
+
+### Content Flow
+
+Each page covers one pipeline stage and ends with a transitional paragraph previewing the next page. Cross-references between pages use relative Markdown links. Diagrams are stored as standalone Mermaid files in the `diagrams/` subdirectory and linked from the narrative pages (not embedded inline).
+
+```mermaid
+flowchart LR
+ P1["Page 1\nData Ingestion"] --> P2["Page 2\nAI Extraction"]
+ P2 --> P3["Page 3\nSignal Scoring"]
+ P3 --> P4["Page 4\nTrend Aggregation"]
+ P4 --> P5["Page 5\nRecommendations"]
+ P5 --> P6["Page 6\nTrading Execution"]
+```
+
+## Components and Interfaces
+
+### Index File (`index.md`)
+
+The index provides:
+- A brief introduction to the deep-dive document series
+- A numbered table of contents linking to all 6 pages
+- A diagrams section linking to all Mermaid diagram files
+- References to existing documentation for additional context
+
+### Narrative Pages (01 through 06)
+
+Each page follows a consistent structure:
+1. **Title and introduction** — what this stage does and why it matters
+2. **Narrative body** — explanatory prose describing the pipeline stage, referencing actual code modules (`services/extractor/main.py`), database tables (`document_impact_records`), Redis queues (`stonks:queue:extraction`), and Pydantic schemas (`ExtractionResult`)
+3. **Diagram references** — links to relevant Mermaid diagram files in `diagrams/`
+4. **Transition** — a closing paragraph that previews the next page
+
+### Mermaid Diagram Files
+
+Each diagram file contains:
+1. A brief title comment
+2. A single Mermaid code block
+3. Service labels include both human-readable names and Python module paths
+4. Queue labels use full Redis key patterns
+5. Database references use exact PostgreSQL table names
+
+Minimum 6 diagrams covering:
+- **Ingestion-to-extraction flow**: Scheduler → Ingestion → Parser → Extractor, with queues and storage
+- **Three-layer signal merging**: Company, Macro, and Competitive layers converging into aggregation
+- **Recommendation generation flow**: Suppression → Eligibility → Thesis → Risk classification
+- **Trading engine decision loop**: Pre-trade checks → Position sizing → Order submission
+- **Weighted signal computation**: Component breakdown of the composite weight formula
+- **Trend accumulation and escalation**: How consecutive signals strengthen trends and escalate actions
+
+### Page Content Mapping
+
+| Page | Primary Code Modules | Key Database Tables | Key Queues |
+|------|---------------------|---------------------|------------|
+| 01 - Ingestion | `services/scheduler/app.py`, `services/ingestion/worker.py`, `services/parser/worker.py` | `documents`, `ingestion_runs`, `document_company_mentions` | `stonks:queue:ingestion`, `stonks:queue:parsing` |
+| 02 - AI Extraction | `services/extractor/main.py`, `services/extractor/client.py`, `services/extractor/prompts.py`, `services/extractor/schemas.py`, `services/extractor/event_classifier.py`, `services/shared/agent_config.py` | `document_intelligence`, `document_impact_records`, `global_events`, `macro_impact_records`, `ai_agents`, `agent_variants` | `stonks:queue:extraction`, `stonks:queue:macro_classification`, `stonks:queue:aggregation` |
+| 03 - Signal Scoring | `services/aggregation/scoring.py` | `document_impact_records`, `macro_impact_records`, `competitive_signal_records`, `risk_configs` | — |
+| 04 - Trend Aggregation | `services/aggregation/worker.py`, `services/aggregation/contradiction.py`, `services/aggregation/projection.py`, `services/aggregation/pattern_matcher.py`, `services/aggregation/signal_propagation.py` | `trend_windows`, `trend_history`, `trend_evidence`, `trend_projections` | `stonks:queue:aggregation`, `stonks:queue:recommendation` |
+| 05 - Recommendations | `services/recommendation/main.py`, `services/recommendation/suppression.py`, `services/recommendation/eligibility.py`, `services/recommendation/thesis_llm.py` | `recommendations`, `recommendation_evidence`, `risk_evaluations` | `stonks:queue:recommendation` |
+| 06 - Trading | `services/trading/engine.py`, `services/trading/position_sizer.py`, `services/trading/circuit_breaker.py`, `services/trading/reserve_pool.py`, `services/trading/risk_tier_controller.py`, `services/trading/stop_loss_manager.py` | `trading_decisions`, `orders`, `positions`, `portfolio_snapshots`, `reserve_pool_ledger`, `risk_tier_history`, `circuit_breaker_events` | `stonks:queue:broker_orders` |
+
+## Data Models
+
+This feature produces only documentation files. There are no new data models, database tables, or schema changes.
+
+The narrative pages will reference existing data models from the codebase:
+
+- **`WeightedSignal`** (`services/aggregation/scoring.py`) — document reference + composite weight + sentiment + impact
+- **`SignalWeight`** (`services/aggregation/scoring.py`) — breakdown of recency, credibility, novelty, confidence gate, market context multiplier
+- **`ScoringConfig`** (`services/aggregation/scoring.py`) — tunable parameters for signal scoring
+- **`ExtractionResult`** / **`CompanyImpact`** (`services/extractor/schemas.py`) — structured JSON output from document extraction
+- **`GlobalEventSchema`** (`services/extractor/event_classifier.py`) — macro event classification output
+- **`TrendSummary`** (`services/shared/schemas.py`) — rolling trend for a ticker across a time window
+- **`Recommendation`** (`services/shared/schemas.py`) — actionable trade recommendation
+- **`TradingDecision`** (`services/trading/engine.py`) — audit record of every trading evaluation
+
+## Error Handling
+
+Since this is a documentation-only deliverable, there is no runtime error handling to design. The primary quality concern is **accuracy** — ensuring that all code module paths, database table names, Redis queue keys, schema field names, and configuration values referenced in the narrative match the actual codebase.
+
+### Accuracy Verification Strategy
+
+1. **Code module paths**: Every module path referenced in the narrative (e.g., `services/aggregation/scoring.py`) must correspond to an existing file in the repository.
+2. **Database table names**: Table names must match those defined in `infra/migrations/` SQL files.
+3. **Redis queue keys**: Queue names must match constants in `services/shared/redis_keys.py`.
+4. **Schema class names**: Pydantic model names must match their definitions in `services/shared/schemas.py` and service-specific schema files.
+5. **Configuration values**: Environment variable names and default values must match `services/shared/config.py` and service-specific configuration.
+
+### Cross-Reference Integrity
+
+All inter-page links (e.g., `[Page 3](03-signal-scoring-and-weighted-signals.md)`) and diagram links (e.g., `[diagram](diagrams/ingestion-to-extraction-flow.md)`) must resolve to files that exist in the deliverable.
+
+## Testing Strategy
+
+**Property-based testing does not apply to this feature.** The deliverable is purely documentation — Markdown narrative pages and Mermaid diagram files. There are no functions, data transformations, or code logic to test.
+
+### Why PBT Does Not Apply
+
+- The output is static Markdown text, not executable code
+- There are no input/output functions to verify properties against
+- There is no data transformation logic that varies with input
+- The quality criteria (narrative coherence, codebase accuracy, cross-reference integrity) are best verified through manual review
+
+### Verification Approach
+
+1. **File existence check**: Verify all 6 page files, the index file, and all diagram files exist at the expected paths
+2. **Link integrity**: Verify all inter-page and diagram links resolve to existing files
+3. **Mermaid syntax**: Verify each diagram file contains valid Mermaid syntax by checking for proper `flowchart` or `graph` declarations
+4. **Codebase reference spot-checks**: Verify a sample of referenced module paths, table names, and queue keys against the actual codebase
+5. **Narrative flow**: Manual review to confirm each page ends with a transition to the next and the overall story is coherent
diff --git a/.kiro/specs/intelligence-pipeline-deep-dive/requirements.md b/.kiro/specs/intelligence-pipeline-deep-dive/requirements.md
new file mode 100644
index 0000000..404376d
--- /dev/null
+++ b/.kiro/specs/intelligence-pipeline-deep-dive/requirements.md
@@ -0,0 +1,155 @@
+# Requirements Document
+
+## Introduction
+
+This specification defines a 6-page narrative deep-dive document (plus separate Mermaid diagram files) that explains the full intelligence-to-decision pipeline in Stonks Oracle. The document targets a technical reader who wants to understand how raw data enters the system, gets processed by AI agents, produces structured signals, accumulates into trend summaries, and ultimately drives autonomous trading decisions. Unlike the existing service reference and API docs, this deliverable is narrative and explanatory — it tells the story of data flowing through the platform end-to-end, referencing actual code modules, database tables, queue names, and schemas from the codebase.
+
+## Glossary
+
+- **Deep_Dive_Document**: The 6-page Markdown document delivered under `docs/intelligence-pipeline-deep-dive/`, consisting of pages 01 through 06 covering the full intelligence-to-decision pipeline.
+- **Mermaid_Diagram_File**: A standalone Markdown file containing a single Mermaid diagram block, stored alongside the narrative pages in `docs/intelligence-pipeline-deep-dive/diagrams/`.
+- **Pipeline**: The end-to-end data flow from external source ingestion through AI extraction, signal aggregation, recommendation generation, and autonomous trading execution.
+- **Signal_Layer**: One of three independent signal sources (Company, Macro, Competitive) that produce `WeightedSignal` objects merged by the Aggregation_Engine.
+- **Aggregation_Engine**: The `services/aggregation/` module that merges weighted signals from all three layers into `TrendSummary` objects across five time windows.
+- **Trading_Engine**: The `services/trading/engine.py` module that polls recommendations and executes autonomous paper trades through a multi-check decision loop.
+- **Extractor**: The `services/extractor/` module that uses Ollama LLM inference to produce structured JSON intelligence from documents.
+- **WeightedSignal**: The `services.aggregation.scoring.WeightedSignal` dataclass that pairs a document reference with a composite aggregation weight.
+- **TrendSummary**: The `services.shared.schemas.TrendSummary` Pydantic model representing a rolling trend for a ticker across a specific time window.
+- **Recommendation**: The `services.shared.schemas.Recommendation` Pydantic model representing an actionable trade recommendation with action, mode, confidence, thesis, and position sizing.
+- **Circuit_Breaker**: The `services/trading/circuit_breaker.py` safety mechanism that halts trading when risk thresholds (daily loss, single-position loss, volatility clustering) are breached.
+
+## Requirements
+
+### Requirement 1: Document Structure and File Organization
+
+**User Story:** As a technical reader, I want the deep-dive organized into clearly separated pages with a consistent structure, so that I can navigate to specific pipeline stages without reading the entire document.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document SHALL consist of exactly 6 Markdown page files named `01-data-ingestion-and-preparation.md` through `06-trading-decisions-and-execution.md`, stored under `docs/intelligence-pipeline-deep-dive/`.
+2. THE Deep_Dive_Document SHALL include an `index.md` file that provides a table of contents linking to all 6 pages and all Mermaid_Diagram_Files.
+3. WHEN a page references a Mermaid diagram, THE Deep_Dive_Document SHALL link to the corresponding Mermaid_Diagram_File stored in `docs/intelligence-pipeline-deep-dive/diagrams/` rather than embedding the diagram inline.
+4. THE Deep_Dive_Document SHALL include a minimum of 4 separate Mermaid_Diagram_Files covering: (a) the ingestion-to-extraction flow, (b) the three signal layers merging into aggregation, (c) the recommendation generation pipeline, and (d) the trading engine decision loop.
+5. WHEN a page references a code module, THE Deep_Dive_Document SHALL use the full Python module path (e.g., `services/extractor/prompts.py`) rather than abbreviated names.
+6. WHEN a page references a database table, THE Deep_Dive_Document SHALL use the exact table name as defined in the PostgreSQL schema (e.g., `document_impact_records`, `trend_windows`).
+7. WHEN a page references a Redis queue, THE Deep_Dive_Document SHALL use the full key pattern as defined in `services/shared/redis_keys.py` (e.g., `stonks:queue:extraction`).
+
+### Requirement 2: Page 1 — Data Ingestion and Preparation
+
+**User Story:** As a technical reader, I want to understand how raw data enters Stonks Oracle and gets prepared for AI processing, so that I can trace the origin of any signal back to its external source.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document page 01 SHALL explain the four categories of input data: news articles (Polygon.io), SEC filings (EDGAR), market data (Polygon.io grouped daily and intraday bars), and macro/geopolitical events (macro news APIs).
+2. THE Deep_Dive_Document page 01 SHALL describe the Scheduler's role in orchestrating ingestion cycles, including cadence polling intervals per source type (`market_api`: 300s, `news_api`: 300s, `filings_api`: 3600s, `macro_news`: 600s), rate limiting, and exponential backoff.
+3. THE Deep_Dive_Document page 01 SHALL describe the Ingestion worker's adapter dispatch pattern, referencing the adapter classes (`PolygonMarketAdapter`, `PolygonNewsAdapter`, `SECEdgarAdapter`, `MacroNewsAdapter`) in `services/ingestion/`.
+4. THE Deep_Dive_Document page 01 SHALL explain content deduplication via Redis content-hash markers (`stonks:dedupe:*` with 24-hour TTL) and raw artifact storage in MinIO buckets (`stonks-raw-market`, `stonks-raw-news`, `stonks-raw-filings`).
+5. THE Deep_Dive_Document page 01 SHALL describe the Parser's role in converting raw HTML/text into normalized documents, including quality scoring with confidence levels (`high`, `medium`, `low`), company mention detection via alias matching, and the routing decision that sends `macro_event` documents to `stonks:queue:macro_classification` instead of `stonks:queue:extraction`.
+6. THE Deep_Dive_Document page 01 SHALL be written in narrative prose style with explanatory paragraphs, not as a reference table or bullet-point list.
+
+### Requirement 3: Page 2 — AI Agent Processing and Structured Extraction
+
+**User Story:** As a technical reader, I want to understand how the AI agents process documents and produce structured JSON output, so that I can evaluate the extraction quality and understand the schema contract.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document page 02 SHALL explain the Document Intelligence Extractor agent (`document-extractor` slug), including its entry point (`services/extractor/main.py` → `services/extractor/client.py`), the system prompt, and the user prompt template built by `build_extraction_prompt()` in `services/extractor/prompts.py`.
+2. THE Deep_Dive_Document page 02 SHALL describe the `ExtractionResult` JSON schema with all fields (summary, companies array with ticker/sentiment/impact_score/impact_horizon/catalyst_type/key_facts/risks/evidence_spans, macro_themes, novelty_score, confidence, extraction_warnings), referencing `services/extractor/schemas.py`.
+3. THE Deep_Dive_Document page 02 SHALL explain the Global Event Classifier agent (`event-classifier` slug), including its entry point (`services/extractor/event_classifier.py`), the `GlobalEvent` output schema with event_types/severity/affected_regions/affected_sectors/affected_commodities/estimated_duration/confidence, and the anti-hallucination rules that prevent classifying company-specific news as macro events.
+4. THE Deep_Dive_Document page 02 SHALL describe the JSON repair pipeline (direct parse → markdown fence stripping → `json-repair` library fallback) and the structural plus semantic validation in `services/extractor/schemas.py`, including retry logic with exponential backoff.
+5. THE Deep_Dive_Document page 02 SHALL explain the `AgentConfigResolver` mechanism (`services/shared/agent_config.py`) that enables hot-swapping models and prompts via the `ai_agents` and `agent_variants` database tables with a 60-second TTL cache.
+6. THE Deep_Dive_Document page 02 SHALL describe how extraction results are persisted to `document_intelligence` (one row per document) and `document_impact_records` (one row per company mention), and how the extractor enqueues aggregation jobs to `stonks:queue:aggregation`.
+7. THE Deep_Dive_Document page 02 SHALL be written in narrative prose style with explanatory paragraphs, not as a reference table or bullet-point list.
+
+### Requirement 4: Page 3 — Signal Scoring and the WeightedSignal Abstraction
+
+**User Story:** As a technical reader, I want to understand how raw extraction output gets transformed into weighted signals for decision making, so that I can reason about why certain documents influence trends more than others.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document page 03 SHALL explain the `WeightedSignal` dataclass (`services/aggregation/scoring.py`) and the composite weight formula: `combined = gate × recency × credibility × (1 + novelty_bonus) × market_context_multiplier`.
+2. THE Deep_Dive_Document page 03 SHALL describe each weight component in detail: confidence gate (threshold 0.2), recency decay (exponential half-life per window: intraday=2h, 1d=12h, 7d=72h, 30d=240h, 90d=720h), source credibility weighting (clamped [0.1, 1.0] with configurable exponent), novelty bonus (up to 25%), and market context multiplier (volatility boost up to 30%, volume surge boost 15%).
+3. THE Deep_Dive_Document page 03 SHALL explain how sentiment labels are mapped to numeric values (+1.0 positive, -1.0 negative, 0.0 neutral/mixed) via `sentiment_to_numeric()` and how the weighted sentiment average is computed across all signals.
+4. THE Deep_Dive_Document page 03 SHALL describe the three signal layers (Company, Macro, Competitive) and how each produces `WeightedSignal` objects that are concatenated into a single list before trend computation, with relative influence controlled by `MACRO_SIGNAL_WEIGHT` (0.3) and `COMPETITIVE_SIGNAL_WEIGHT` (0.2).
+5. THE Deep_Dive_Document page 03 SHALL explain the runtime toggle mechanism for macro and competitive layers via the `risk_configs` database table, including graceful degradation when a layer is disabled or fails.
+6. THE Deep_Dive_Document page 03 SHALL be written in narrative prose style with explanatory paragraphs, not as a reference table or bullet-point list.
+
+### Requirement 5: Page 4 — Trend Aggregation and Accumulating Signals
+
+**User Story:** As a technical reader, I want to understand how the aggregation engine merges multiple signals — including consecutive signals suggesting the same direction — to produce trend summaries that drive grander decisions, so that I can see how accumulating bearish or bullish evidence escalates the system's response.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document page 04 SHALL explain how the Aggregation_Engine (`services/aggregation/worker.py`) computes `TrendSummary` objects across five time windows (intraday, 1d, 7d, 30d, 90d) by fetching impact records, macro impacts, and competitive signals for a ticker.
+2. THE Deep_Dive_Document page 04 SHALL describe the trend direction derivation rules: bullish (avg_sentiment ≥ 0.15), bearish (avg_sentiment ≤ -0.15), mixed (contradiction > 0.10 and |avg_sentiment| < 0.30), neutral (otherwise), referencing `derive_trend_direction()` in `services/aggregation/worker.py`.
+3. THE Deep_Dive_Document page 04 SHALL explain contradiction detection (`services/aggregation/contradiction.py`), including sentiment disagreement analysis and catalyst-level disagreement, and how the contradiction score (minority_weight / total_weight) penalizes trend confidence.
+4. THE Deep_Dive_Document page 04 SHALL describe how consecutive signals in the same direction accumulate to strengthen trend_strength and confidence, explaining the evidence ranking mechanism (`rank_evidence()`) that uses composite scoring (weight, impact, recency, confidence) and the confidence computation that rewards unique source count (caps at 15 sources for 0.8 contribution) and signal agreement (log₂ scaling, saturates around 7 unique sources).
+5. THE Deep_Dive_Document page 04 SHALL explain how accumulating bearish signals across multiple documents and time windows escalate the system's response — from a neutral hold to a bearish sell recommendation — and conversely how accumulating bullish signals escalate from watch to buy, using the trend strength and confidence thresholds from the eligibility rules.
+6. THE Deep_Dive_Document page 04 SHALL describe trend projections (`services/aggregation/projection.py`), including macro decay, momentum, driving factors, and divergence detection.
+7. THE Deep_Dive_Document page 04 SHALL describe persistence to `trend_windows` (upserted each cycle), `trend_history` (time-series snapshots), `trend_evidence` (per-document rankings), and `trend_projections`.
+8. THE Deep_Dive_Document page 04 SHALL be written in narrative prose style with explanatory paragraphs, not as a reference table or bullet-point list.
+
+### Requirement 6: Page 5 — Recommendation Generation and Signal-to-Action Translation
+
+**User Story:** As a technical reader, I want to understand how trend summaries are translated into actionable recommendations with risk classification and thesis generation, so that I can see the decision logic between aggregated intelligence and trading actions.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document page 05 SHALL explain the data quality suppression layer (`services/recommendation/suppression.py`), including the six suppression checks (extraction confidence < 0.40, evidence staleness > 168h, source diversity < 1, extraction failure rate > 50%, valid document count < 2, data quality score < 0.30) and the safety suppressions for macro-only and pattern-only trend shifts.
+2. THE Deep_Dive_Document page 05 SHALL describe the eligibility evaluation (`services/recommendation/eligibility.py`), including gate checks (confidence ≥ 0.35, strength ≥ 0.10, contradiction ≤ 0.60, evidence ≥ 2, direction ≠ neutral), action mapping (BUY/SELL for strength ≥ 0.25, HOLD for weaker directional signals, WATCH otherwise), and mode escalation (informational → paper_eligible → live_eligible based on confidence and evidence thresholds).
+3. THE Deep_Dive_Document page 05 SHALL explain position sizing computation from signal quality: base 1% + confidence × strength scaling up to 10%, with contradiction penalty, evidence count penalty, and max loss percentage scaling.
+4. THE Deep_Dive_Document page 05 SHALL describe the two-layer thesis generation: deterministic thesis assembly from trend data, and optional LLM rewrite via the `thesis-rewriter` agent (`services/recommendation/thesis_llm.py`) for trading-eligible recommendations.
+5. THE Deep_Dive_Document page 05 SHALL explain risk classification (low/moderate/high/very_high) based on contradiction score, confidence, evidence count, and mode.
+6. THE Deep_Dive_Document page 05 SHALL describe persistence to `recommendations`, `recommendation_evidence`, and `risk_evaluations` tables.
+7. THE Deep_Dive_Document page 05 SHALL be written in narrative prose style with explanatory paragraphs, not as a reference table or bullet-point list.
+
+### Requirement 7: Page 6 — Trading Engine Decisions and Execution
+
+**User Story:** As a technical reader, I want to understand how the trading engine uses aggregated trend data to make buy/sell/hold decisions, including position sizing, risk evaluation, and circuit breakers, so that I can trace any trade back to its intelligence origin.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document page 06 SHALL explain the Trading_Engine decision loop (`services/trading/engine.py`), including the five concurrent async tasks: decision loop (60s polling), stop-loss monitor, performance loop, risk tier scheduler, and rebalance scheduler.
+2. THE Deep_Dive_Document page 06 SHALL describe the pre-trade check sequence in order: circuit breaker check, trading window check, confidence gate (risk-tier minimum), deduplication, declining positions check, and max open positions check, explaining that the first failure short-circuits the evaluation.
+3. THE Deep_Dive_Document page 06 SHALL explain position sizing (`services/trading/position_sizer.py`), including confidence-based scaling with sample-size-dampened agreement scoring, risk tier adjustment (conservative/moderate/aggressive with specific parameter differences), correlation-aware diversification, sector exposure reduction, earnings proximity adjustment, and the absolute position cap.
+4. THE Deep_Dive_Document page 06 SHALL describe the Circuit_Breaker mechanism (`services/trading/circuit_breaker.py`), including the three trigger types (daily_loss with emergency drawdown threshold, single_position loss with ticker cooldown, volatility with stop-loss clustering detection), cooldown computation, and Redis state tracking (`stonks:trading:circuit_breaker:*`).
+5. THE Deep_Dive_Document page 06 SHALL explain the reserve pool mechanism (`services/trading/reserve_pool.py`): profit siphoning (default 20%), high-water mark rebalancing (30% threshold), emergency liquidation, and ledger tracking in `reserve_pool_ledger`.
+6. THE Deep_Dive_Document page 06 SHALL describe risk tier auto-adjustment (`services/trading/risk_tier_controller.py`), including the evaluation criteria (Sharpe ratio, drawdown, win rate) and the three tier configurations with their parameter differences (min confidence, max position %, stop-loss ATR multiplier, reward/risk ratio, max sector %, max portfolio heat).
+7. THE Deep_Dive_Document page 06 SHALL explain the order submission flow: `TradingDecision` persistence to `trading_decisions`, order job enqueue to `stonks:queue:broker_orders`, broker adapter risk evaluation, Alpaca paper trading submission, and the full audit trail from signal to broker response.
+8. THE Deep_Dive_Document page 06 SHALL be written in narrative prose style with explanatory paragraphs, not as a reference table or bullet-point list.
+
+### Requirement 8: Mermaid Diagram Quality and Separation
+
+**User Story:** As a technical reader, I want architecture diagrams in separate files that I can render independently, so that I can use them in presentations or embed them in other documents.
+
+#### Acceptance Criteria
+
+1. WHEN a Mermaid_Diagram_File is created, THE Deep_Dive_Document SHALL store the diagram in a standalone Markdown file under `docs/intelligence-pipeline-deep-dive/diagrams/` with a descriptive filename (e.g., `ingestion-to-extraction-flow.md`).
+2. THE Deep_Dive_Document SHALL include at least 4 Mermaid_Diagram_Files: one for the ingestion-to-extraction pipeline, one for the three-layer signal merging, one for the recommendation generation flow, and one for the trading engine decision loop.
+3. WHEN a Mermaid diagram references a service, THE Mermaid_Diagram_File SHALL label the service with both its human-readable name and its Python module path (e.g., `Extractor\nservices/extractor/main.py`).
+4. WHEN a Mermaid diagram references a queue, THE Mermaid_Diagram_File SHALL use the full Redis key pattern (e.g., `stonks:queue:extraction`).
+5. WHEN a Mermaid diagram references a database table, THE Mermaid_Diagram_File SHALL use the exact PostgreSQL table name.
+
+### Requirement 9: Narrative Style and Cross-Referencing
+
+**User Story:** As a technical reader, I want the document to read as a coherent narrative rather than a reference manual, so that I can build a mental model of the full pipeline without jumping between disconnected sections.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document SHALL use narrative prose with explanatory paragraphs as the primary writing style, reserving tables and bullet lists for structured data summaries only.
+2. WHEN a page references content covered in a different page, THE Deep_Dive_Document SHALL include a Markdown link to the relevant page and section.
+3. THE Deep_Dive_Document SHALL include transitional paragraphs at the end of each page that preview what the next page covers, creating a continuous narrative flow.
+4. THE Deep_Dive_Document SHALL reference the existing documentation where appropriate (e.g., `docs/services.md`, `docs/ai-agents.md`, `docs/architecture-data-pipeline.md`, `docs/llm-to-trade-pipeline.md`) for readers who want deeper reference-level detail.
+5. IF a concept is introduced for the first time, THEN THE Deep_Dive_Document SHALL provide a brief inline explanation before using the concept in subsequent discussion.
+
+### Requirement 10: Codebase Accuracy
+
+**User Story:** As a developer, I want the document to reference actual code modules, database tables, and queue names from the codebase, so that I can use the document as a reliable guide when navigating the source code.
+
+#### Acceptance Criteria
+
+1. THE Deep_Dive_Document SHALL reference code modules using paths that exist in the repository (e.g., `services/aggregation/scoring.py`, `services/trading/circuit_breaker.py`, `services/shared/schemas.py`).
+2. THE Deep_Dive_Document SHALL reference database tables using names that match the PostgreSQL schema as defined in `infra/migrations/`.
+3. THE Deep_Dive_Document SHALL reference Redis queue names using the constants defined in `services/shared/redis_keys.py` (e.g., `QUEUE_EXTRACTION`, `QUEUE_AGGREGATION`, `QUEUE_RECOMMENDATION`, `QUEUE_BROKER`).
+4. THE Deep_Dive_Document SHALL reference Pydantic schema classes using their actual class names from `services/shared/schemas.py` (e.g., `DocumentIntelligence`, `TrendSummary`, `Recommendation`, `GlobalEventSchema`, `CompanyImpact`).
+5. THE Deep_Dive_Document SHALL reference configuration environment variables using the exact names defined in `services/shared/config.py` and the service-specific configuration sections.
diff --git a/.kiro/specs/intelligence-pipeline-deep-dive/tasks.md b/.kiro/specs/intelligence-pipeline-deep-dive/tasks.md
new file mode 100644
index 0000000..2eab822
--- /dev/null
+++ b/.kiro/specs/intelligence-pipeline-deep-dive/tasks.md
@@ -0,0 +1,35 @@
+# Tasks — Intelligence Pipeline Deep Dive
+
+## Task 1: Create directory structure and index file
+- [x] 1.1 Create `docs/intelligence-pipeline-deep-dive/` directory and `docs/intelligence-pipeline-deep-dive/diagrams/` subdirectory
+- [x] 1.2 Create `docs/intelligence-pipeline-deep-dive/index.md` with table of contents linking to all 6 pages and all diagram files, plus references to existing docs (`docs/services.md`, `docs/ai-agents.md`, `docs/architecture-data-pipeline.md`, `docs/llm-to-trade-pipeline.md`)
+
+## Task 2: Create Mermaid diagram files
+- [x] 2.1 Create `docs/intelligence-pipeline-deep-dive/diagrams/ingestion-to-extraction-flow.md` — flowchart from Scheduler through Ingestion, Parser, to Extractor with all queues (`stonks:queue:ingestion`, `stonks:queue:parsing`, `stonks:queue:extraction`, `stonks:queue:macro_classification`), storage (MinIO buckets, PostgreSQL tables), and service module paths
+- [x] 2.2 Create `docs/intelligence-pipeline-deep-dive/diagrams/three-layer-signal-merging.md` — flowchart showing Company signals (`document_impact_records`), Macro signals (`macro_impact_records`), and Competitive signals (`competitive_signal_records`) each producing `WeightedSignal` objects that merge into the Aggregation engine (`services/aggregation/worker.py`)
+- [x] 2.3 Create `docs/intelligence-pipeline-deep-dive/diagrams/weighted-signal-computation.md` — diagram showing the composite weight formula components: confidence gate, recency decay, source credibility, novelty bonus, and market context multiplier
+- [x] 2.4 Create `docs/intelligence-pipeline-deep-dive/diagrams/trend-accumulation-escalation.md` — diagram showing how consecutive signals accumulate across time windows to escalate from neutral → watch → hold → buy/sell decisions
+- [x] 2.5 Create `docs/intelligence-pipeline-deep-dive/diagrams/recommendation-generation-flow.md` — flowchart from TrendSummary through data quality suppression, eligibility evaluation, thesis generation, risk classification, to recommendation persistence
+- [x] 2.6 Create `docs/intelligence-pipeline-deep-dive/diagrams/trading-engine-decision-loop.md` — flowchart showing the pre-trade check sequence (circuit breaker → trading window → confidence gate → dedup → declining positions → max positions), position sizing, and order submission to `stonks:queue:broker_orders`
+
+## Task 3: Write Page 1 — Data Ingestion and Preparation
+- [x] 3.1 Write `docs/intelligence-pipeline-deep-dive/01-data-ingestion-and-preparation.md` covering: four input data categories (Polygon news, SEC EDGAR filings, Polygon market data, macro news APIs), Scheduler cadence polling (market_api: 300s, news_api: 300s, filings_api: 3600s, macro_news: 600s) with rate limiting and backoff, Ingestion worker adapter dispatch (`PolygonMarketAdapter`, `PolygonNewsAdapter`, `SECEdgarAdapter`, `MacroNewsAdapter`), content deduplication via Redis (`stonks:dedupe:*` with 24h TTL), raw artifact storage in MinIO (`stonks-raw-market`, `stonks-raw-news`, `stonks-raw-filings`), Parser role (HTML normalization, quality scoring, company mention detection, routing `macro_event` docs to `stonks:queue:macro_classification`). Written in narrative prose with links to diagrams and transition to Page 2.
+
+## Task 4: Write Page 2 — AI Agent Processing and Structured Extraction
+- [x] 4.1 Write `docs/intelligence-pipeline-deep-dive/02-ai-agent-processing-and-extraction.md` covering: Document Intelligence Extractor agent (`document-extractor` slug, `services/extractor/main.py` → `services/extractor/client.py`, system prompt, `build_extraction_prompt()` in `services/extractor/prompts.py`), `ExtractionResult` JSON schema with all fields, Global Event Classifier agent (`event-classifier` slug, `services/extractor/event_classifier.py`, `GlobalEvent` schema, anti-hallucination rules), JSON repair pipeline (direct parse → fence stripping → `json-repair` fallback), structural + semantic validation in `services/extractor/schemas.py`, `AgentConfigResolver` mechanism (`services/shared/agent_config.py`, `ai_agents`/`agent_variants` tables, 60s TTL cache), persistence to `document_intelligence` and `document_impact_records`, aggregation job enqueue. Written in narrative prose with links to diagrams and transition to Page 3.
+
+## Task 5: Write Page 3 — Signal Scoring and the WeightedSignal Abstraction
+- [x] 5.1 Write `docs/intelligence-pipeline-deep-dive/03-signal-scoring-and-weighted-signals.md` covering: `WeightedSignal` dataclass (`services/aggregation/scoring.py`), composite weight formula (`combined = gate × recency × credibility × (1 + novelty_bonus) × market_context_multiplier`), each component in detail (confidence gate threshold 0.2, recency decay half-lives per window, source credibility clamped [0.1, 1.0], novelty bonus up to 25%, market context volatility boost up to 30% and volume surge boost 15%), sentiment mapping via `sentiment_to_numeric()`, weighted sentiment average computation, three signal layers (Company, Macro weight 0.3, Competitive weight 0.2), runtime toggle via `risk_configs` table. Written in narrative prose with links to diagrams and transition to Page 4.
+
+## Task 6: Write Page 4 — Trend Aggregation and Accumulating Signals
+- [x] 6.1 Write `docs/intelligence-pipeline-deep-dive/04-trend-aggregation-and-accumulating-signals.md` covering: Aggregation engine computing TrendSummary across 5 windows (intraday, 1d, 7d, 30d, 90d), trend direction rules (bullish ≥ 0.15, bearish ≤ -0.15, mixed, neutral), contradiction detection (`services/aggregation/contradiction.py`, minority_weight/total_weight), evidence ranking (`rank_evidence()` composite scoring), confidence computation (unique source count caps at 15, log₂ scaling saturates at 7 sources), how consecutive same-direction signals accumulate to escalate decisions (neutral → watch → hold → buy/sell), trend projections (`services/aggregation/projection.py`, macro decay, momentum, divergence detection), persistence to `trend_windows`, `trend_history`, `trend_evidence`, `trend_projections`. Written in narrative prose with links to diagrams and transition to Page 5.
+
+## Task 7: Write Page 5 — Recommendation Generation and Signal-to-Action Translation
+- [x] 7.1 Write `docs/intelligence-pipeline-deep-dive/05-recommendation-generation.md` covering: data quality suppression (`services/recommendation/suppression.py`, 6 checks: extraction confidence < 0.40, staleness > 168h, source diversity < 1, failure rate > 50%, valid docs < 2, quality score < 0.30, plus macro-only and pattern-only safety), eligibility evaluation (`services/recommendation/eligibility.py`, gate checks, action mapping BUY/SELL/HOLD/WATCH, mode escalation informational/paper_eligible/live_eligible), position sizing (base 1% + confidence × strength up to 10%, contradiction and evidence penalties), thesis generation (deterministic + optional LLM rewrite via `thesis-rewriter` agent), risk classification (low/moderate/high/very_high), persistence to `recommendations`, `recommendation_evidence`, `risk_evaluations`. Written in narrative prose with links to diagrams and transition to Page 6.
+
+## Task 8: Write Page 6 — Trading Engine Decisions and Execution
+- [x] 8.1 Write `docs/intelligence-pipeline-deep-dive/06-trading-decisions-and-execution.md` covering: Trading engine decision loop (`services/trading/engine.py`, 5 concurrent tasks: decision loop 60s, stop-loss monitor, performance loop, risk tier scheduler, rebalance scheduler), pre-trade check sequence (circuit breaker → trading window → confidence gate → dedup → declining positions → max positions), position sizing (`services/trading/position_sizer.py`, confidence scaling, risk tier adjustment, correlation diversification, sector exposure, earnings proximity, absolute cap), circuit breaker (`services/trading/circuit_breaker.py`, daily_loss, single_position, volatility triggers, cooldown, Redis state), reserve pool (`services/trading/reserve_pool.py`, profit siphoning 20%, high-water mark 30%, emergency liquidation), risk tier auto-adjustment (`services/trading/risk_tier_controller.py`, Sharpe/drawdown/win-rate evaluation, conservative/moderate/aggressive tiers), order submission flow (TradingDecision → `stonks:queue:broker_orders` → broker adapter → Alpaca). Written in narrative prose with links to diagrams.
+
+## Task 9: Update index and verify cross-references
+- [x] 9.1 Update `docs/intelligence-pipeline-deep-dive/index.md` to ensure all page links and diagram links are correct and all files exist
+- [x] 9.2 Verify all inter-page links within narrative pages resolve correctly and all diagram references point to existing files
diff --git a/.kiro/specs/sanitized-pipeline-docs/.config.kiro b/.kiro/specs/sanitized-pipeline-docs/.config.kiro
new file mode 100644
index 0000000..cae7df8
--- /dev/null
+++ b/.kiro/specs/sanitized-pipeline-docs/.config.kiro
@@ -0,0 +1 @@
+{"specId": "e6d189b2-5861-4e24-954f-5e254246a910", "workflowType": "requirements-first", "specType": "feature"}
diff --git a/.kiro/specs/sanitized-pipeline-docs/design.md b/.kiro/specs/sanitized-pipeline-docs/design.md
new file mode 100644
index 0000000..bd59b0d
--- /dev/null
+++ b/.kiro/specs/sanitized-pipeline-docs/design.md
@@ -0,0 +1,341 @@
+# Design Document: Sanitized Pipeline Documentation
+
+## Overview
+
+This design specifies the process and structure for producing a sanitized version of the 6-page intelligence pipeline deep dive documentation. The sanitized docs transform the existing `docs/intelligence-pipeline-deep-dive/` content into domain-neutral equivalents stored at `docs/sanitized-pipeline-deep-dive/`, stripping all financial, market, and trading language while preserving every engineering detail — algorithms, formulas, architectural patterns, queue topologies, database schemas, code module references, and Mermaid diagrams.
+
+The deliverable is a documentation-only transformation. No application code, database schemas, or infrastructure changes are involved. The output is Markdown files and Mermaid diagram files that mirror the original structure with domain-neutral framing.
+
+**Key design decision**: The sanitization is a manual content transformation guided by a defined terminology map. Each source file is read, transformed according to the mapping rules, and written to the output directory. The original files remain untouched.
+
+### Source Material
+
+The source documentation at `docs/intelligence-pipeline-deep-dive/` consists of:
+
+| File | Content |
+|------|---------|
+| `index.md` | Table of contents, introduction, diagram links, related docs |
+| `01-data-ingestion-and-preparation.md` | Scheduler, ingestion worker, deduplication, parser |
+| `02-ai-agent-processing-and-extraction.md` | Document extractor, event classifier, JSON repair, validation |
+| `03-signal-scoring-and-weighted-signals.md` | Composite weight formula, three signal layers, sentiment mapping |
+| `04-trend-aggregation-and-accumulating-signals.md` | Time windows, trend direction, contradiction, evidence ranking, confidence |
+| `05-recommendation-generation.md` | Suppression, eligibility, position sizing, thesis, risk classification |
+| `06-trading-decisions-and-execution.md` | Trading engine, pre-trade checks, circuit breakers, broker adapter |
+| `diagrams/ingestion-to-extraction-flow.md` | Mermaid flowchart: scheduler → ingestion → parser → extractor |
+| `diagrams/three-layer-signal-merging.md` | Mermaid flowchart: three signal layers → aggregation |
+| `diagrams/weighted-signal-computation.md` | Mermaid flowchart: composite weight formula breakdown |
+| `diagrams/trend-accumulation-escalation.md` | Mermaid flowchart: time windows → escalation path |
+| `diagrams/recommendation-generation-flow.md` | Mermaid flowchart: suppression → eligibility → thesis → risk |
+| `diagrams/trading-engine-decision-loop.md` | Mermaid flowchart: pre-trade checks → position sizing → order submission |
+
+## Architecture
+
+### Output File Organization
+
+The sanitized docs mirror the source structure with sanitized filenames:
+
+```
+docs/sanitized-pipeline-deep-dive/
+├── index.md
+├── 01-data-ingestion-and-preparation.md
+├── 02-ai-agent-processing-and-extraction.md
+├── 03-signal-scoring-and-weighted-signals.md
+├── 04-trend-aggregation-and-accumulating-signals.md
+├── 05-recommendation-generation.md
+├── 06-decision-execution.md
+└── diagrams/
+ ├── ingestion-to-extraction-flow.md
+ ├── three-layer-signal-merging.md
+ ├── weighted-signal-computation.md
+ ├── trend-accumulation-escalation.md
+ ├── recommendation-generation-flow.md
+ └── decision-engine-loop.md
+```
+
+**Filename changes from source:**
+- `06-trading-decisions-and-execution.md` → `06-decision-execution.md` (removes "trading")
+- `diagrams/trading-engine-decision-loop.md` → `diagrams/decision-engine-loop.md` (removes "trading")
+- All other filenames are already domain-neutral and remain unchanged
+
+### Transformation Process
+
+The sanitization follows a three-pass approach for each file:
+
+1. **Terminology pass**: Apply the terminology map to replace all financial/trading terms with domain-neutral equivalents. This covers inline text, headings, table cells, code blocks, and Mermaid diagram labels.
+2. **Reference pass**: Update all internal cross-references to point to sanitized filenames (e.g., `06-trading-decisions-and-execution.md` → `06-decision-execution.md`, `trading-engine-decision-loop.md` → `decision-engine-loop.md`). Remove or neutralize references to external financial docs (e.g., links to `../llm-to-trade-pipeline.md` become neutral descriptions).
+3. **Narrative pass**: Reframe example scenarios, inline illustrations, and narrative framing to use domain-neutral language. This pass handles context-dependent replacements that a simple find-and-replace cannot catch — e.g., "a bearish article about AAPL" becomes "a negative-sentiment article about Entity-A".
+
+### Content Flow
+
+The sanitized docs preserve the same page-to-page narrative flow as the originals:
+
+```mermaid
+flowchart LR
+ P1["Page 1\nData Ingestion"] --> P2["Page 2\nAI Extraction"]
+ P2 --> P3["Page 3\nSignal Scoring"]
+ P3 --> P4["Page 4\nTrend Aggregation"]
+ P4 --> P5["Page 5\nRecommendations"]
+ P5 --> P6["Page 6\nDecision Execution"]
+```
+
+## Components and Interfaces
+
+### Terminology Map
+
+The core of the sanitization is a defined mapping from financial/trading terms to domain-neutral equivalents. The map is applied consistently across all files.
+
+#### System and Provider Names
+
+| Source Term | Sanitized Replacement |
+|-------------|----------------------|
+| Stonks Oracle / stonks | the platform / the system |
+| Polygon.io / Polygon | external data provider / data source API |
+| SEC EDGAR / SEC / EFTS | public records API / regulatory filings source |
+| Alpaca / AlpacaBrokerAdapter | execution adapter / external execution API |
+| Wall Street | (removed or reframed) |
+
+#### Trading and Financial Actions
+
+| Source Term | Sanitized Replacement |
+|-------------|----------------------|
+| buy | act |
+| sell | defer |
+| hold | monitor |
+| watch | observe |
+| trading engine | decision execution engine |
+| paper trading / paper_eligible | simulation mode / simulation_eligible |
+| live trading / live_eligible | live execution mode / production_eligible |
+| trade / trading (as action) | decision / execution |
+| order (broker order) | execution request |
+| pre-trade checks | pre-execution checks |
+
+#### Financial Concepts
+
+| Source Term | Sanitized Replacement |
+|-------------|----------------------|
+| portfolio | resource pool / allocation pool |
+| portfolio allocation | resource allocation |
+| portfolio heat | pool exposure |
+| portfolio snapshots | pool snapshots |
+| position sizing | commitment sizing / resource allocation |
+| position (open position) | commitment / active commitment |
+| stop-loss | risk threshold / loss limit |
+| take-profit | gain target |
+| bullish | positive / favorable |
+| bearish | negative / unfavorable |
+| stock ticker / ticker symbol | entity identifier |
+| stock market | (removed or reframed) |
+| earnings / earnings call / earnings report | performance report / periodic disclosure |
+| 10-K / 10-Q / 8-K | regulatory filing types |
+| SEC filings | regulatory filings |
+| broker / broker API | execution adapter / execution API |
+| P&L | gain/loss |
+| Sharpe ratio | risk-adjusted return ratio |
+| drawdown | peak-to-trough decline |
+| win rate | success rate |
+
+#### Ticker Symbols and Company Names
+
+| Source Term | Sanitized Replacement |
+|-------------|----------------------|
+| AAPL / Apple | Entity-A |
+| TSLA / Tesla | Entity-B |
+| NVDA / NVIDIA | Entity-C |
+| XOM | Entity-D |
+| META | Entity-E |
+| Any other ticker | Entity-{letter} or "tracked entity" |
+
+#### Redis Keys
+
+| Source Pattern | Sanitized Pattern |
+|----------------|-------------------|
+| `stonks:queue:*` | `app:queue:*` |
+| `stonks:dedupe:*` | `app:dedupe:*` |
+| `stonks:ratelimit:*` | `app:ratelimit:*` |
+| `stonks:trading:circuit_breaker:*` | `app:execution:circuit_breaker:*` |
+| `stonks:dedupe:trading:*` | `app:dedupe:execution:*` |
+
+#### MinIO Buckets
+
+| Source Bucket | Sanitized Bucket |
+|---------------|-----------------|
+| `stonks-raw-market` | `app-raw-data` |
+| `stonks-raw-news` | `app-raw-content` |
+| `stonks-raw-filings` | `app-raw-filings` |
+| `stonks-normalized` | `app-normalized` |
+| `stonks-llm-prompts` | `app-llm-prompts` |
+| `stonks-llm-results` | `app-llm-results` |
+
+#### Database Tables
+
+| Source Table | Sanitized Table |
+|-------------|----------------|
+| `trading_decisions` | `execution_decisions` |
+| `portfolio_snapshots` | `pool_snapshots` |
+| `portfolio_pct` (column) | `allocation_pct` |
+
+All other table names (`documents`, `document_intelligence`, `trend_windows`, `recommendations`, etc.) are already domain-neutral and remain unchanged.
+
+#### Adapter and Source Type Names
+
+| Source Term | Sanitized Replacement |
+|-------------|----------------------|
+| `PolygonNewsAdapter` | `ExternalNewsAdapter` |
+| `PolygonMarketAdapter` | `ExternalDataAdapter` |
+| `SECEdgarAdapter` | `RegulatoryFilingsAdapter` |
+| `AlpacaBrokerAdapter` | `ExecutionAdapter` |
+| `broker` (source_type) | `execution_api` |
+| `market_api` (source_type) | `data_api` |
+| `filings_api` (source_type) | `filings_api` (unchanged — already neutral) |
+
+### Preserved Engineering Terms
+
+The following terms are explicitly preserved because they describe engineering patterns, not financial concepts:
+
+- **circuit breaker** — engineering safety pattern for rate limiting and cascading failure prevention
+- **exponential backoff** — retry pattern
+- **adapter pattern** — software design pattern (only the domain-specific adapter *names* are sanitized)
+- **signal** — used in signal processing and scoring context
+- **trend**, **sentiment**, **confidence**, **contradiction**, **evidence** — data analysis terms
+- **recency decay**, **credibility weight**, **novelty bonus** — scoring algorithm terms
+- **weighted sentiment average** — mathematical computation term
+
+### Preserved Technical Content
+
+All of the following are preserved verbatim (with only the terminology map applied to embedded financial terms):
+
+- Composite signal scoring formula: `combined = gate × recency × credibility × (1 + novelty_bonus) × market_context_multiplier`
+- Confidence computation formula with log₂ scaling and four components
+- Weighted sentiment average formula
+- All threshold values, configuration parameters, and numeric constants
+- All Markdown table structures containing technical parameters
+- All code module path references (e.g., `services/aggregation/scoring.py`)
+- Three-layer signal architecture with weight ratios (1.0, 0.3, 0.2)
+- Contradiction detection algorithm and evidence ranking methodology
+- All PostgreSQL table structures and column descriptions (with sanitized names where needed)
+- All Redis queue patterns and operations (`rpush`/`lpop`/`blpop`)
+- All MinIO storage patterns (with sanitized bucket names)
+- Ollama as the LLM inference provider
+
+### Index Page Reframing
+
+The sanitized `index.md` describes the system as an "AI-driven intelligence-to-decision pipeline" that:
+1. Ingests data from multiple external data sources
+2. Extracts structured intelligence via NLP/LLM
+3. Scores and weights signals
+4. Aggregates trends across time windows
+5. Generates recommendations with quality gates
+6. Executes decisions autonomously with safety mechanisms
+
+References to "Stonks Oracle" are replaced with "the platform" or "the system". References to financial-specific APIs (Polygon.io, SEC EDGAR) are replaced with neutral descriptions. The "Related Documentation" section links are updated to use neutral descriptions or removed if they reference financial-specific content.
+
+### Page 06 Reframing
+
+Page 06 undergoes the most extensive reframing since it covers the trading engine. Key changes:
+- Title: "Decision Execution" instead of "Trading Decisions and Execution"
+- "Trading engine" → "decision execution engine"
+- "Pre-trade checks" → "pre-execution checks"
+- "Broker adapter" / "Alpaca" → "execution adapter" / "external execution API"
+- "Paper trading" → "simulation mode"
+- "Live trading" → "live execution mode"
+- "Portfolio" → "resource pool" / "allocation pool"
+- "Position" → "commitment" / "active commitment"
+- "Stop-loss" → "risk threshold"
+- "Take-profit" → "gain target"
+- All order submission language reframed as "execution request submission"
+
+### Diagram Sanitization
+
+Each Mermaid diagram file receives the same terminology map treatment:
+- Node labels containing financial terms are replaced
+- Queue name labels (`stonks:queue:*` → `app:queue:*`)
+- Bucket name labels (`stonks-raw-market` → `app-raw-data`)
+- Table name labels (`trading_decisions` → `execution_decisions`)
+- Adapter names in node labels
+- Subgraph titles containing financial terms
+- The `trading-engine-decision-loop.md` diagram is renamed to `decision-engine-loop.md`
+
+Mermaid syntax, node relationships, subgraph structures, and flow directions are preserved exactly.
+
+## Data Models
+
+This feature produces only documentation files. There are no new data models, database tables, or schema changes.
+
+The sanitized narrative pages reference the same data models as the originals, with terminology-mapped names where applicable:
+
+- **`WeightedSignal`** — document reference + composite weight + sentiment + impact (unchanged)
+- **`SignalWeight`** — breakdown of recency, credibility, novelty, confidence gate, market context multiplier (unchanged)
+- **`TrendSummary`** — rolling trend for an entity across a time window (unchanged)
+- **`Recommendation`** — actionable decision recommendation (reframed from "trade recommendation")
+- **`execution_decisions`** table — audit record of every decision evaluation (sanitized from `trading_decisions`)
+- **`pool_snapshots`** table — resource pool state snapshots (sanitized from `portfolio_snapshots`)
+
+
+## Correctness Properties
+
+*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
+
+The sanitized documentation set has one key universal property: the complete absence of financial/trading terminology across all output files. This is well-suited to property-based testing because the property must hold for *every* file in the output set, and the banned term list is large enough that systematic checking across all files provides high-value coverage.
+
+### Property 1: Banned Financial Terminology Exclusion
+
+*For any* file in the sanitized documentation set (`docs/sanitized-pipeline-deep-dive/`), the file content shall not contain any term from the comprehensive banned financial terminology list. The banned list includes: stock ticker symbols (AAPL, TSLA, NVDA, XOM, META, and all 50 tracked tickers), company names used as financial examples (Apple, Tesla, NVIDIA), trading action labels (buy, sell, hold, watch as action labels — BUY, SELL, HOLD, WATCH in uppercase), financial system terms (trading engine, paper trading, live trading, paper_eligible, live_eligible, portfolio, portfolio allocation, portfolio heat, portfolio snapshots, broker, Alpaca, broker adapter, broker API, stock market, Wall Street, bullish, bearish, position sizing, stop-loss), financial event terms (SEC EDGAR, SEC filings, 10-K, 10-Q, 8-K, earnings, earnings call, earnings report), provider names (Polygon.io, Polygon), system names (Stonks Oracle, stonks), and infrastructure patterns containing financial terms (stonks: prefix in Redis keys, stonks- prefix in MinIO buckets, trading_decisions table name, portfolio_snapshots table name).
+
+**Validates: Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, 6.2, 7.1, 7.2, 7.3, 8.1, 8.2**
+
+## Error Handling
+
+Since this is a documentation-only deliverable, there is no runtime error handling to design. The primary quality concerns are:
+
+### Accuracy of Terminology Replacement
+
+Every financial/trading term must be replaced with its domain-neutral equivalent. Missing a single instance of "stonks" in a Redis key pattern or "AAPL" in an example scenario would violate the sanitization requirements. The terminology map defined in the Components section serves as the authoritative reference.
+
+### Preservation of Technical Content
+
+The sanitization must not accidentally remove or alter engineering content. Key risks:
+- **Formula corruption**: The composite weight formula contains `market_context_multiplier` — the word "market" must not be blindly replaced since it's part of a technical variable name
+- **Code path corruption**: Module paths like `services/trading/engine.py` contain "trading" — these paths reference actual files and must be preserved as-is (the code files are not being renamed)
+- **Table name corruption**: Database table names like `trading_decisions` need sanitization in narrative text but the actual SQL/code references to the original table names should be handled carefully
+
+**Design decision**: Code module paths (e.g., `services/trading/engine.py`) are preserved exactly as they appear in the source, since they reference actual files in the repository. Only narrative references to concepts (e.g., "the trading engine") are sanitized. Variable names within formulas and code blocks are preserved. Database table names are sanitized in narrative descriptions and table listings, but inline code references note the sanitized name.
+
+### Cross-Reference Integrity
+
+All internal links must resolve to files that exist in the sanitized output:
+- Page-to-page links must use sanitized filenames
+- Diagram links must use sanitized diagram filenames
+- No links should point back to the source `docs/intelligence-pipeline-deep-dive/` directory
+
+## Testing Strategy
+
+### Why Limited PBT Applies
+
+This is a documentation-only deliverable — the output is static Markdown files, not executable code with functions and data transformations. However, one universal property (banned term exclusion) is well-suited to property-based testing because it must hold across all files and involves checking a large set of terms against file content.
+
+Most other requirements (structural checks, content preservation, narrative reframing) are better verified through example-based tests and manual review.
+
+### Property-Based Tests
+
+- **Library**: Hypothesis (Python, already in the project)
+- **Configuration**: `@settings(max_examples=100)`
+- **Property 1 implementation**: Generate random selections from the banned term list and random file selections from the sanitized docs, verify the term does not appear in the file content. Alternatively, exhaustively check all banned terms against all files (since the file set is small and fixed, this is more practical as an exhaustive example-based test).
+
+**Practical note**: Given the small, fixed file set (14 files), the banned term exclusion property is most practically implemented as an exhaustive check — iterate all files × all banned terms — rather than a randomized property test. This provides complete coverage rather than probabilistic coverage.
+
+### Example-Based Tests
+
+1. **File structure verification**: Verify all expected files exist at the correct paths
+2. **Cross-reference integrity**: Parse all sanitized files, extract markdown links, verify they resolve to existing sanitized files
+3. **Mermaid syntax validation**: Verify each diagram file contains valid Mermaid `flowchart` declarations
+4. **Technical content preservation**: Spot-check that key formulas, threshold values, and code module paths are present in the sanitized docs
+5. **Terminology replacement verification**: Spot-check that key replacements appear (e.g., "decision execution engine" replaces "trading engine")
+6. **Index page framing**: Verify the index describes the system as an "AI-driven intelligence-to-decision pipeline"
+7. **Database table sanitization**: Verify `execution_decisions` appears where `trading_decisions` was, and `pool_snapshots` where `portfolio_snapshots` was
+
+### Manual Review
+
+- Narrative coherence and readability of the sanitized content
+- Consistency of domain-neutral framing across all pages
+- Quality of example scenario replacements (e.g., "bearish article about AAPL" → "negative-sentiment article about Entity-A")
+- Preservation of page-to-page transition flow
diff --git a/.kiro/specs/sanitized-pipeline-docs/requirements.md b/.kiro/specs/sanitized-pipeline-docs/requirements.md
new file mode 100644
index 0000000..65e613c
--- /dev/null
+++ b/.kiro/specs/sanitized-pipeline-docs/requirements.md
@@ -0,0 +1,202 @@
+# Requirements Document
+
+## Introduction
+
+This feature produces a sanitized version of the existing 6-page intelligence pipeline deep dive documentation (`docs/intelligence-pipeline-deep-dive/`) for use in a work presentation. The sanitized version strips all financial, market, and trading language — stock tickers, buy/sell/hold actions, portfolio allocation, broker APIs, and domain-specific framing — and reframes the content as a general-purpose AI decision intelligence pipeline. The sanitized docs are stored as a separate doc group under `docs/sanitized-pipeline-deep-dive/`, preserving the original documents untouched. All engineering depth — algorithms, formulas, architectural patterns, queue topologies, database schemas, code module references, and Mermaid diagrams — is preserved. Only the domain-specific framing changes.
+
+## Glossary
+
+- **Source_Docs**: The original 6-page documentation set at `docs/intelligence-pipeline-deep-dive/`, including `index.md`, pages `01` through `06`, and the `diagrams/` subdirectory containing 6 Mermaid diagram files.
+- **Sanitized_Docs**: The output documentation set at `docs/sanitized-pipeline-deep-dive/`, mirroring the structure of Source_Docs with all financial/market/trading language replaced by domain-neutral equivalents.
+- **Sanitization_Engine**: The process (manual or automated) that transforms Source_Docs into Sanitized_Docs by applying the terminology mapping and content reframing rules defined in this document.
+- **Terminology_Map**: The defined set of financial/market/trading terms and their domain-neutral replacements used by the Sanitization_Engine.
+- **Entity_Identifier**: The domain-neutral replacement for stock ticker symbols (e.g., AAPL, TSLA) in Sanitized_Docs.
+- **Decision_Term**: A domain-neutral action term (act, defer, monitor, observe) that replaces trading actions (buy, sell, hold, watch) in Sanitized_Docs.
+- **Decision_Execution_Engine**: The domain-neutral name for the trading engine in Sanitized_Docs.
+- **Execution_Adapter**: The domain-neutral name for broker adapters and broker API references in Sanitized_Docs.
+- **Allocation_Pool**: The domain-neutral name for portfolio references in Sanitized_Docs.
+- **Commitment_Sizing**: The domain-neutral name for position sizing in Sanitized_Docs.
+
+---
+
+## Requirements
+
+### Requirement 1: Separate Output Directory
+
+**User Story:** As a presenter, I want the sanitized docs stored in a separate directory from the originals, so that the original documentation remains untouched and both versions coexist.
+
+#### Acceptance Criteria
+
+1. THE Sanitization_Engine SHALL write all output files to `docs/sanitized-pipeline-deep-dive/`.
+2. THE Sanitization_Engine SHALL NOT modify, overwrite, or delete any file under `docs/intelligence-pipeline-deep-dive/`.
+3. THE Sanitized_Docs SHALL contain an `index.md` file at the root of `docs/sanitized-pipeline-deep-dive/`.
+4. THE Sanitized_Docs SHALL contain a `diagrams/` subdirectory under `docs/sanitized-pipeline-deep-dive/`.
+
+---
+
+### Requirement 2: Mirror the 6-Page Structure
+
+**User Story:** As a presenter, I want the sanitized docs to mirror the same 6-page structure as the originals, so that readers familiar with the original can navigate the sanitized version identically.
+
+#### Acceptance Criteria
+
+1. THE Sanitized_Docs SHALL contain exactly 6 numbered page files matching the naming pattern of Source_Docs: `01-*.md` through `06-*.md`.
+2. THE Sanitized_Docs SHALL contain an `index.md` with a table of contents linking to all 6 pages and all diagrams, mirroring the structure of the Source_Docs index.
+3. THE Sanitized_Docs SHALL contain one Mermaid diagram file in `diagrams/` for each diagram file present in `docs/intelligence-pipeline-deep-dive/diagrams/`.
+4. WHEN a Source_Docs page contains internal cross-references to other pages or diagrams, THE Sanitized_Docs equivalent page SHALL contain corresponding cross-references pointing to the Sanitized_Docs versions of those pages and diagrams.
+5. THE Sanitized_Docs page filenames SHALL use sanitized titles (e.g., `06-decision-execution.md` instead of `06-trading-decisions-and-execution.md`).
+
+---
+
+### Requirement 3: Strip Financial and Trading Terminology
+
+**User Story:** As a presenter, I want all financial, market, and trading language removed from the sanitized docs, so that the presentation focuses on engineering without revealing the financial domain.
+
+#### Acceptance Criteria
+
+1. THE Sanitized_Docs SHALL NOT contain any stock ticker symbols (e.g., AAPL, TSLA, NVDA, XOM, META).
+2. THE Sanitized_Docs SHALL NOT contain the trading action terms "buy", "sell", "hold", or "watch" when used as system action labels or decision outputs.
+3. THE Sanitized_Docs SHALL NOT contain the terms "trading engine", "paper trading", "live trading", "paper_eligible", or "live_eligible".
+4. THE Sanitized_Docs SHALL NOT contain the terms "portfolio", "portfolio allocation", "portfolio heat", or "portfolio snapshots" when referring to the resource management domain concept.
+5. THE Sanitized_Docs SHALL NOT contain references to "broker", "Alpaca", "broker adapter", or "broker API".
+6. THE Sanitized_Docs SHALL NOT contain the terms "stock market", "Wall Street", "bullish", "bearish", "position sizing" (as a financial concept label), or "stop-loss" (as a financial concept label).
+7. THE Sanitized_Docs SHALL NOT contain company names used as financial examples (e.g., "Apple", "Tesla", "NVIDIA" when used in a stock/market context).
+8. THE Sanitized_Docs SHALL NOT contain the terms "SEC EDGAR", "SEC filings", "10-K", "10-Q", "8-K", "earnings", "earnings call", or "earnings report" as domain-specific financial references.
+9. THE Sanitized_Docs SHALL NOT contain references to "Polygon.io" or "Polygon" as a financial data provider name.
+10. THE Sanitized_Docs SHALL NOT contain the term "Stonks Oracle" or "stonks" as a system name.
+
+---
+
+### Requirement 4: Apply Domain-Neutral Terminology Mapping
+
+**User Story:** As a presenter, I want consistent domain-neutral replacements for all stripped terms, so that the sanitized docs read coherently as a general-purpose AI decision intelligence pipeline.
+
+#### Acceptance Criteria
+
+1. WHEN the Source_Docs use "stock ticker" or specific ticker symbols, THE Sanitized_Docs SHALL use "entity identifier" or "tracked entity".
+2. WHEN the Source_Docs use "buy/sell/hold/watch" as action labels, THE Sanitized_Docs SHALL use "act/defer/monitor/observe" or equivalent neutral decision terms.
+3. WHEN the Source_Docs use "trading engine", THE Sanitized_Docs SHALL use "decision execution engine" or "action engine".
+4. WHEN the Source_Docs use "portfolio", THE Sanitized_Docs SHALL use "resource pool" or "allocation pool".
+5. WHEN the Source_Docs use "broker" or "Alpaca", THE Sanitized_Docs SHALL use "execution adapter" or "external execution API".
+6. WHEN the Source_Docs use "paper trading", THE Sanitized_Docs SHALL use "simulation mode" or "dry-run mode".
+7. WHEN the Source_Docs use "live trading", THE Sanitized_Docs SHALL use "live execution mode" or "production mode".
+8. WHEN the Source_Docs use "bullish" or "bearish", THE Sanitized_Docs SHALL use "positive" or "negative" (or "favorable"/"unfavorable").
+9. WHEN the Source_Docs use "position sizing", THE Sanitized_Docs SHALL use "resource allocation" or "commitment sizing".
+10. WHEN the Source_Docs use "stop-loss", THE Sanitized_Docs SHALL use "risk threshold" or "loss limit".
+11. WHEN the Source_Docs use "Stonks Oracle" or "stonks", THE Sanitized_Docs SHALL use a neutral system name such as "the platform" or "the system".
+12. WHEN the Source_Docs use "SEC EDGAR" or "SEC filings", THE Sanitized_Docs SHALL use "regulatory filings source" or "public records API".
+13. WHEN the Source_Docs use "Polygon.io" or "Polygon", THE Sanitized_Docs SHALL use "external data provider" or "data source API".
+14. WHEN the Source_Docs use "earnings" as a catalyst type or event, THE Sanitized_Docs SHALL use "performance report" or "periodic disclosure".
+15. THE Sanitized_Docs SHALL apply the Terminology_Map consistently across all 6 pages, the index, and all diagram files.
+
+---
+
+### Requirement 5: Preserve Engineering and Technical Depth
+
+**User Story:** As a presenter, I want all engineering concepts, algorithms, formulas, and architectural details preserved, so that the sanitized docs demonstrate the technical sophistication of the system.
+
+#### Acceptance Criteria
+
+1. THE Sanitized_Docs SHALL preserve all references to Redis queue patterns, including queue names and `rpush`/`lpop`/`blpop` operations.
+2. THE Sanitized_Docs SHALL preserve all references to PostgreSQL tables, including table names and column descriptions.
+3. THE Sanitized_Docs SHALL preserve all references to MinIO buckets and storage patterns.
+4. THE Sanitized_Docs SHALL preserve all references to Ollama as the LLM inference provider.
+5. THE Sanitized_Docs SHALL preserve the composite signal scoring formula: `combined = gate × recency × credibility × (1 + novelty_bonus) × market_context_multiplier`.
+6. THE Sanitized_Docs SHALL preserve the confidence computation formula with log₂ scaling and its four components (unique source count, average extraction credibility, signal agreement with sample-size dampening, contradiction penalty).
+7. THE Sanitized_Docs SHALL preserve the weighted sentiment average formula: `weighted_avg = Σ(combined_weight × impact_score × sentiment_value) / Σ(combined_weight × impact_score)`.
+8. THE Sanitized_Docs SHALL preserve all code module path references (e.g., `services/aggregation/scoring.py`, `services/recommendation/eligibility.py`).
+9. THE Sanitized_Docs SHALL preserve the three-layer signal architecture, renaming the layers with domain-neutral labels (e.g., "Entity-Specific Signals", "Environmental Signals", "Relational Signals") while retaining the weight ratios (1.0, 0.3, 0.2).
+10. THE Sanitized_Docs SHALL preserve all threshold values, configuration parameters, and numeric constants (e.g., confidence gate of 0.2, recency half-lives per window, eligibility thresholds).
+11. THE Sanitized_Docs SHALL preserve all Markdown table structures containing technical parameters and thresholds.
+12. THE Sanitized_Docs SHALL preserve the contradiction detection algorithm, evidence ranking methodology, and trend projection computation.
+
+---
+
+### Requirement 6: Sanitize Mermaid Diagrams
+
+**User Story:** As a presenter, I want the Mermaid diagrams sanitized with the same terminology mapping as the narrative pages, so that diagrams and text are consistent.
+
+#### Acceptance Criteria
+
+1. THE Sanitized_Docs SHALL contain one sanitized Mermaid diagram file for each of the 6 diagram files in Source_Docs.
+2. WHEN a Source_Docs diagram contains financial/trading terminology (e.g., "trading engine", "buy/sell", "paper_eligible", "bullish/bearish", ticker symbols), THE corresponding Sanitized_Docs diagram SHALL use the same domain-neutral replacements defined in the Terminology_Map.
+3. THE Sanitized_Docs diagrams SHALL preserve all Mermaid syntax, node relationships, subgraph structures, and flow directions from the Source_Docs diagrams.
+4. THE Sanitized_Docs diagrams SHALL preserve all code module path references and service names within diagram nodes.
+5. THE Sanitized_Docs diagram filenames SHALL use sanitized names where the original names contain financial terms (e.g., `decision-engine-loop.md` instead of `trading-engine-decision-loop.md`).
+
+---
+
+### Requirement 7: Sanitize Redis Key and Queue Name References
+
+**User Story:** As a presenter, I want Redis key patterns and queue names sanitized where they contain financial terms, so that even infrastructure-level references are domain-neutral.
+
+#### Acceptance Criteria
+
+1. WHEN a Source_Docs Redis queue name contains "stonks" (e.g., `stonks:queue:ingestion`), THE Sanitized_Docs SHALL replace "stonks" with a neutral prefix (e.g., `app:queue:ingestion`).
+2. WHEN a Source_Docs Redis key pattern contains "trading" (e.g., `stonks:queue:broker_orders`, `stonks:trading:circuit_breaker:*`), THE Sanitized_Docs SHALL replace the trading-specific segment with a neutral equivalent (e.g., `app:queue:execution_orders`, `app:execution:circuit_breaker:*`).
+3. THE Sanitized_Docs SHALL apply Redis key sanitization consistently across all narrative pages and diagram files.
+
+---
+
+### Requirement 8: Sanitize MinIO Bucket Name References
+
+**User Story:** As a presenter, I want MinIO bucket names sanitized where they contain financial terms, so that storage references are domain-neutral.
+
+#### Acceptance Criteria
+
+1. WHEN a Source_Docs MinIO bucket name contains "stonks" (e.g., `stonks-raw-market`, `stonks-raw-news`, `stonks-normalized`), THE Sanitized_Docs SHALL replace "stonks" with a neutral prefix (e.g., `app-raw-data`, `app-raw-content`, `app-normalized`).
+2. THE Sanitized_Docs SHALL apply MinIO bucket name sanitization consistently across all narrative pages and diagram files.
+
+---
+
+### Requirement 9: Sanitize Database Table and Column References Where Needed
+
+**User Story:** As a presenter, I want database table and column names that contain obvious financial terms sanitized, while preserving the overall schema structure.
+
+#### Acceptance Criteria
+
+1. WHEN a Source_Docs database table name contains "trading" (e.g., `trading_decisions`), THE Sanitized_Docs SHALL use a neutral equivalent (e.g., `execution_decisions`).
+2. WHEN a Source_Docs database table or column references "portfolio" (e.g., `portfolio_snapshots`, `portfolio_pct`), THE Sanitized_Docs SHALL use a neutral equivalent (e.g., `pool_snapshots`, `allocation_pct`).
+3. THE Sanitized_Docs SHALL preserve all other database table names that do not contain financial-specific terms (e.g., `documents`, `document_intelligence`, `trend_windows`, `recommendations`).
+4. THE Sanitized_Docs SHALL apply database reference sanitization consistently across all narrative pages.
+
+---
+
+### Requirement 10: Sanitize Example Scenarios and Inline References
+
+**User Story:** As a presenter, I want all inline examples, scenario walkthroughs, and narrative references sanitized, so that no financial context leaks through illustrative content.
+
+#### Acceptance Criteria
+
+1. WHEN a Source_Docs page uses a specific company name or ticker in an example scenario (e.g., "a bearish article about AAPL"), THE Sanitized_Docs SHALL replace the reference with a generic entity (e.g., "a negative-sentiment article about Entity-A").
+2. WHEN a Source_Docs page describes a financial event as an example (e.g., "earnings miss", "tariff announcement affecting XOM"), THE Sanitized_Docs SHALL reframe the example using domain-neutral language (e.g., "a negative performance disclosure", "a regulatory policy change affecting Entity-B").
+3. WHEN a Source_Docs page references market-specific concepts in narrative flow (e.g., "markets move fast", "trading volume", "intraday swings"), THE Sanitized_Docs SHALL reframe using neutral language (e.g., "conditions change rapidly", "activity volume", "short-term fluctuations").
+4. THE Sanitized_Docs SHALL preserve the logical structure and teaching purpose of all example scenarios while removing the financial framing.
+
+---
+
+### Requirement 11: Preserve Acceptable Engineering Terms
+
+**User Story:** As a presenter, I want general engineering terms that happen to overlap with financial language preserved when they describe engineering patterns, so that the technical accuracy is maintained.
+
+#### Acceptance Criteria
+
+1. THE Sanitized_Docs SHALL preserve the term "circuit breaker" when it describes the engineering safety pattern (rate limiting, cascading failure prevention).
+2. THE Sanitized_Docs SHALL preserve the term "exponential backoff" and all retry/backoff patterns.
+3. THE Sanitized_Docs SHALL preserve all adapter pattern references (the software design pattern), renaming only the domain-specific adapter names (e.g., "AlpacaBrokerAdapter" becomes a neutral name).
+4. THE Sanitized_Docs SHALL preserve the term "signal" as used in the signal processing and scoring context.
+5. THE Sanitized_Docs SHALL preserve the terms "trend", "sentiment", "confidence", "contradiction", and "evidence" as used in the data analysis context.
+
+---
+
+### Requirement 12: Reframe the System Narrative
+
+**User Story:** As a presenter, I want the overall system narrative reframed as a general-purpose AI decision intelligence pipeline, so that the presentation tells a coherent story without financial context.
+
+#### Acceptance Criteria
+
+1. THE Sanitized_Docs index page SHALL describe the system as an "AI-driven intelligence-to-decision pipeline" that ingests data from multiple sources, extracts structured intelligence via NLP/LLM, scores and weights signals, aggregates trends across time windows, generates recommendations with quality gates, and executes decisions autonomously with safety mechanisms.
+2. THE Sanitized_Docs page 01 SHALL describe data ingestion from "multiple external data sources" rather than from financial-specific APIs.
+3. THE Sanitized_Docs page 06 SHALL describe "autonomous decision execution with safety mechanisms" rather than "trading decisions and execution".
+4. WHEN the Source_Docs conclusion references the "intelligence-to-decision pipeline in Stonks Oracle", THE Sanitized_Docs conclusion SHALL reference the "intelligence-to-decision pipeline" without a financial system name.
+5. THE Sanitized_Docs SHALL maintain the narrative flow where each page ends with a transition to the next page, preserving the end-to-end story structure.
diff --git a/.kiro/specs/sanitized-pipeline-docs/tasks.md b/.kiro/specs/sanitized-pipeline-docs/tasks.md
new file mode 100644
index 0000000..f48468f
--- /dev/null
+++ b/.kiro/specs/sanitized-pipeline-docs/tasks.md
@@ -0,0 +1,47 @@
+# Tasks — Sanitized Pipeline Documentation
+
+## Task 1: Create Output Directory and Index Page
+
+- [x] 1.1 Create the `docs/sanitized-pipeline-deep-dive/` directory and `diagrams/` subdirectory
+- [x] 1.2 Create `docs/sanitized-pipeline-deep-dive/index.md` with sanitized content: replace "Stonks Oracle" with "the platform", replace Polygon.io/SEC EDGAR references with neutral descriptions, update all page links to use sanitized filenames (e.g., `06-decision-execution.md`), update diagram links to use sanitized names (e.g., `decision-engine-loop.md`), describe the system as an "AI-driven intelligence-to-decision pipeline", and update or remove the Related Documentation section to use neutral descriptions
+
+## Task 2: Sanitize Page 01 — Data Ingestion and Preparation
+
+- [x] 2.1 Create `docs/sanitized-pipeline-deep-dive/01-data-ingestion-and-preparation.md` by transforming the source page: replace "Stonks Oracle" with "the platform", replace "Polygon.io" with "external data provider", replace "SEC EDGAR"/"EFTS" with "public records API"/"regulatory filings source", replace "AlpacaBrokerAdapter" with "ExecutionAdapter", replace adapter class names (PolygonNewsAdapter → ExternalNewsAdapter, PolygonMarketAdapter → ExternalDataAdapter, SECEdgarAdapter → RegulatoryFilingsAdapter), replace all `stonks:` Redis key prefixes with `app:`, replace MinIO bucket names (stonks-raw-market → app-raw-data, stonks-raw-news → app-raw-content, stonks-raw-filings → app-raw-filings, stonks-normalized → app-normalized), replace ticker symbols (AAPL → Entity-A, etc.) and company names with generic entities, replace "broker" source_type with "execution_api", replace "SEC" references with "regulatory filings", replace "10-K"/"10-Q"/"8-K" with "regulatory filing types", replace "earnings" with "performance report", sanitize example paths (e.g., `news_api/AAPL/...` → `news_api/Entity-A/...`), update cross-references to use sanitized filenames, and preserve all engineering content (queue operations, table structures, quality scoring formula, code module paths)
+
+## Task 3: Sanitize Page 02 — AI Agent Processing and Extraction
+
+- [x] 3.1 Create `docs/sanitized-pipeline-deep-dive/02-ai-agent-processing-and-extraction.md` by transforming the source page: replace "Stonks Oracle" references, replace financial document type references (SEC filings → regulatory filings, earnings transcripts → performance transcripts), replace "financial document analyst" role description with "document analyst", replace ticker symbols and company names in examples (AAPL, TSLA, NVDA, XOM, META → Entity-A through Entity-E), replace "bearish"/"bullish" with "negative"/"positive", replace "earnings" catalyst type references with "performance_report", replace "stock ticker" with "entity identifier", replace "market implications" with neutral language, replace `stonks:queue:*` Redis keys with `app:queue:*`, replace MinIO bucket names (stonks-llm-prompts → app-llm-prompts, stonks-llm-results → app-llm-results, stonks-normalized → app-normalized), replace "tariff announcement affecting XOM" example with neutral equivalent, update cross-references, and preserve all engineering content (JSON repair pipeline, validation logic, AgentConfigResolver, Ollama references, code module paths, schema field descriptions)
+
+## Task 4: Sanitize Page 03 — Signal Scoring and Weighted Signals
+
+- [x] 4.1 Create `docs/sanitized-pipeline-deep-dive/03-signal-scoring-and-weighted-signals.md` by transforming the source page: replace "bullish"/"bearish" with "positive"/"negative" throughout, replace "trading recommendations" with "decision recommendations", replace ticker examples (AAPL, NVDA) with Entity-A/Entity-C, replace "market context" variable references carefully (preserve `market_context_multiplier` as a technical variable name but sanitize narrative references to "market conditions" → "environmental conditions"), replace "trading volume" with "activity volume", replace `stonks:queue:*` Redis keys with `app:queue:*`, replace "bullish_pct > bearish_pct" with "positive_pct > negative_pct" in signal propagation description, update cross-references, and preserve all engineering content (composite weight formula, recency decay formula, half-life tables, credibility weight computation, novelty bonus formula, weighted sentiment average formula, three-layer architecture with weight ratios 1.0/0.3/0.2, all threshold values and configuration parameters)
+
+## Task 5: Sanitize Page 04 — Trend Aggregation and Accumulating Signals
+
+- [x] 5.1 Create `docs/sanitized-pipeline-deep-dive/04-trend-aggregation-and-accumulating-signals.md` by transforming the source page: replace "bullish"/"bearish" with "positive"/"negative" in trend direction descriptions and TrendDirection enum values, replace "trading recommendations" with "decision recommendations", replace "BULLISH_THRESHOLD"/"BEARISH_THRESHOLD" with "POSITIVE_THRESHOLD"/"NEGATIVE_THRESHOLD", replace "paper_eligible"/"live_eligible" with "simulation_eligible"/"production_eligible", replace "paper trading"/"live trading" with "simulation mode"/"live execution mode", replace "buy"/"sell"/"hold"/"watch" action labels with "act"/"defer"/"monitor"/"observe", replace "trading_decisions" table with "execution_decisions", replace "portfolio" references, replace ticker examples (AAPL) with Entity-A, replace "earnings miss" example with "negative performance disclosure", replace `stonks:queue:*` Redis keys with `app:queue:*`, update cross-references, and preserve all engineering content (five time windows, trend direction derivation thresholds, contradiction detection algorithm, evidence ranking, confidence computation formula with log₂ scaling, trend projection computation, all persistence tables)
+
+## Task 6: Sanitize Page 05 — Recommendation Generation
+
+- [x] 6.1 Create `docs/sanitized-pipeline-deep-dive/05-recommendation-generation.md` by transforming the source page: replace "buy"/"sell"/"hold"/"watch" action labels with "act"/"defer"/"monitor"/"observe", replace "BUY"/"SELL"/"HOLD"/"WATCH" with "ACT"/"DEFER"/"MONITOR"/"OBSERVE", replace "paper_eligible"/"live_eligible" with "simulation_eligible"/"production_eligible", replace "paper trading"/"live trading" with "simulation mode"/"live execution mode", replace "trading engine" with "decision execution engine", replace "portfolio" with "resource pool"/"allocation pool", replace "portfolio_pct" with "allocation_pct", replace "position sizing" with "commitment sizing", replace "position" (as financial position) with "commitment", replace "stop-loss" with "risk threshold", replace "trading-eligible" with "execution-eligible", replace "trade" (as noun/verb) with "decision"/"execution", replace ticker examples (AAPL) with Entity-A, replace "earnings" catalyst references with "performance_report", replace `stonks:queue:*` Redis keys with `app:queue:*`, replace "broker adapter" with "execution adapter", replace "Alpaca" with "external execution API", update cross-references to use sanitized filenames (06-decision-execution.md), and preserve all engineering content (suppression thresholds, eligibility gates, position sizing formulas, thesis generation logic, risk classification computation, all persistence tables)
+
+## Task 7: Sanitize Page 06 — Decision Execution
+
+- [x] 7.1 Create `docs/sanitized-pipeline-deep-dive/06-decision-execution.md` by transforming the source page: change title to "Decision Execution", replace "trading engine" with "decision execution engine" throughout, replace "TradingEngine" class references with "DecisionEngine" in narrative (preserve code module path `services/trading/engine.py`), replace "trade"/"trading" with "decision"/"execution" in narrative, replace "pre-trade checks" with "pre-execution checks", replace "buy"/"sell" action labels with "act"/"defer", replace "paper trading"/"paper_eligible" with "simulation mode"/"simulation_eligible", replace "live trading"/"live_eligible" with "live execution mode"/"production_eligible", replace "broker"/"Alpaca" with "execution adapter"/"external execution API", replace "AlpacaBrokerAdapter" with "ExecutionAdapter" in narrative, replace "portfolio" with "resource pool"/"allocation pool", replace "portfolio heat" with "pool exposure", replace "portfolio_snapshots" with "pool_snapshots", replace "position"/"positions" (financial) with "commitment"/"commitments", replace "position sizing"/"PositionSizer" with "commitment sizing" in narrative, replace "stop-loss" with "risk threshold", replace "take-profit" with "gain target", replace "P&L" with "gain/loss", replace "Sharpe ratio" with "risk-adjusted return ratio", replace "win rate" with "success rate", replace "drawdown" with "peak-to-trough decline", replace "trading_decisions" table with "execution_decisions", replace `stonks:queue:broker_orders` with `app:queue:execution_orders`, replace `stonks:trading:circuit_breaker:*` with `app:execution:circuit_breaker:*`, replace `stonks:dedupe:trading:*` with `app:dedupe:execution:*`, replace all other `stonks:` Redis key prefixes with `app:`, replace "paper-api.alpaca.markets" with "execution-api.example.com", replace "Polygon API" with "data source API", replace ticker examples with Entity-{letter}, replace "earnings" references with "performance report"/"periodic disclosure", update cross-references to use sanitized filenames, update the Conclusion section to remove "Stonks Oracle" and financial framing, and preserve all engineering content (5 concurrent async tasks, circuit breaker algorithm, reserve pool logic, risk tier parameters table, position sizing pipeline, order submission flow, all code module paths, all threshold values)
+
+## Task 8: Sanitize Mermaid Diagrams
+
+- [x] 8.1 Create `docs/sanitized-pipeline-deep-dive/diagrams/ingestion-to-extraction-flow.md` by transforming the source diagram: replace `stonks:queue:*` with `app:queue:*`, replace MinIO bucket names (stonks-raw-market → app-raw-data, stonks-raw-news → app-raw-content, stonks-raw-filings → app-raw-filings, stonks-normalized → app-normalized), replace adapter names in node labels (PolygonMarketAdapter → ExternalDataAdapter, PolygonNewsAdapter → ExternalNewsAdapter, SECEdgarAdapter → RegulatoryFilingsAdapter, MacroNewsAdapter unchanged, WebScrapeAdapter unchanged), replace "AlpacaBrokerAdapter" if present, and preserve all Mermaid syntax, node relationships, subgraph structures, flow directions, and code module paths
+- [x] 8.2 Create `docs/sanitized-pipeline-deep-dive/diagrams/three-layer-signal-merging.md` by transforming the source diagram: replace `stonks:queue:*` with `app:queue:*`, replace "bullish_pct > bearish_pct" if present, and preserve all Mermaid syntax and structure
+- [x] 8.3 Create `docs/sanitized-pipeline-deep-dive/diagrams/weighted-signal-computation.md` by copying the source diagram with minimal changes (content is already domain-neutral — only replace any `stonks:` references if present), preserving all Mermaid syntax and structure
+- [x] 8.4 Create `docs/sanitized-pipeline-deep-dive/diagrams/trend-accumulation-escalation.md` by transforming the source diagram: replace "BULLISH"/"BEARISH" with "POSITIVE"/"NEGATIVE", replace "BUY / SELL" with "ACT / DEFER", replace "paper_eligible"/"live_eligible" if present, and preserve all Mermaid syntax and structure
+- [x] 8.5 Create `docs/sanitized-pipeline-deep-dive/diagrams/recommendation-generation-flow.md` by transforming the source diagram: replace `stonks:queue:*` with `app:queue:*`, replace "BUY"/"SELL"/"HOLD"/"WATCH" with "ACT"/"DEFER"/"MONITOR"/"OBSERVE", replace "paper_eligible"/"live_eligible" with "simulation_eligible"/"production_eligible", replace "portfolio" with "allocation pool", and preserve all Mermaid syntax and structure
+- [x] 8.6 Create `docs/sanitized-pipeline-deep-dive/diagrams/decision-engine-loop.md` (renamed from trading-engine-decision-loop.md) by transforming the source diagram: replace "Trading Engine" with "Decision Execution Engine", replace `stonks:queue:broker_orders` with `app:queue:execution_orders`, replace `stonks:dedupe:trading:*` with `app:dedupe:execution:*`, replace `stonks:trading:circuit_breaker:*` with `app:execution:circuit_breaker:*`, replace "buy, sell" with "act, defer", replace "paper_eligible, live_eligible" with "simulation_eligible, production_eligible", replace "Alpaca paper trading" with "external execution API (simulation)", replace "portfolio" references with "resource pool"/"allocation pool", replace "Portfolio heat" with "Pool exposure", replace "portfolio_snapshots" with "pool_snapshots", replace "trading_decisions" with "execution_decisions", replace "Sharpe ratio" with "risk-adjusted return ratio", replace "drawdown" with "peak-to-trough decline", replace "win rate" with "success rate", replace "P&L" with "gain/loss", and preserve all Mermaid syntax, node relationships, subgraph structures, flow directions, and code module paths
+
+## Task 9: Verification and Cross-Reference Integrity
+
+- [x] 9.1 Verify all sanitized files exist at the expected paths: index.md, 6 numbered pages (01-06), and 6 diagram files in diagrams/
+- [x] 9.2 Verify no sanitized file contains any banned financial term: scan all files for ticker symbols (AAPL, TSLA, NVDA, XOM, META), company names (Apple, Tesla, NVIDIA as financial references), system names (Stonks Oracle, stonks), provider names (Polygon.io, Polygon, SEC EDGAR, Alpaca), financial terms (trading engine, paper trading, live trading, paper_eligible, live_eligible, portfolio, broker, bullish, bearish, position sizing, stop-loss, stock market, Wall Street, earnings, 10-K, 10-Q, 8-K), and infrastructure patterns (stonks: prefix, stonks- prefix, trading_decisions, portfolio_snapshots)
+- [x] 9.3 Verify all internal cross-references resolve: parse all markdown links in sanitized files, confirm each link target exists in the sanitized output directory
+- [x] 9.4 Verify key engineering content is preserved: check that the composite weight formula, confidence computation formula, weighted sentiment average formula, three-layer weight ratios (1.0, 0.3, 0.2), and key threshold values (confidence gate 0.2, eligibility confidence 0.35) appear in the sanitized docs
+- [x] 9.5 Verify source files are unmodified: confirm that no files under `docs/intelligence-pipeline-deep-dive/` were changed
diff --git a/.kiro/steering/development-process.md b/.kiro/steering/development-process.md
index edbfa4b..57c52f7 100644
--- a/.kiro/steering/development-process.md
+++ b/.kiro/steering/development-process.md
@@ -30,18 +30,16 @@
- Ruff config: `ruff.toml` with `known-first-party = ["services"]` for consistent import sorting
- Pre-existing test failures (not regressions): `test_extractor_prompts.py`, `test_extractor_schemas.py`, `test_filings_adapter.py`, `test_ollama_client.py`
-## CI/CD — GitHub Actions
-- Workflow: `.github/workflows/build.yml`
-- Triggers on push to `main` and PRs
-- Jobs:
- - `lint-and-test`: ruff lint + pytest + frontend vitest (Node 24)
- - `build-services`: matrix build of all Python services → GHCR
- - `build-dashboard`: frontend/Dockerfile → GHCR (TypeScript strict mode — catches unused imports)
- - `build-superset`: docker/Dockerfile.superset → GHCR
+## CI/CD — Woodpecker CI (Gitea) → GitHub promotion
+- Woodpecker pipelines in `.woodpecker/` — triggered by push to `main` on Gitea
+- Push to Gitea: `git push gitea main`
+- Gitea remote: `http://admin:@10.1.1.12:30300/admin/stonks-oracle.git`
+- Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
+- ArgoCD watches Gitea `main` and auto-syncs beta/paper/live stages
+- **Do NOT push directly to GitHub** — GitHub is the promotion target after CI passes
+- Once Woodpecker builds and tests pass, code is promoted to GitHub (`git push origin main`)
- CI handles all image builds and pushes — do NOT manually docker push
-- Check CI: `gh run list -L 3`
-- Re-run failed: `gh run rerun --failed`
-- View failure logs: `gh run view --log-failed`
+- Check Woodpecker CI status from the Gitea web UI or Woodpecker dashboard
## Deploy
- Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1)
@@ -74,7 +72,9 @@ Ingestion jobs MUST include `source_id`, `source_type`, `ticker`, `company_id`,
## Git Conventions
- Commit after each completed phase task
- Commit message format: `feat:`, `fix:`, `phase N:` prefix
-- Push to `main` triggers CI
+- Always push to Gitea: `git push gitea main`
+- Do NOT push to GitHub (`origin`) directly — GitHub is the promotion target after CI passes
+- ArgoCD syncs from Gitea automatically
## Code Style
- Python 3.12, type hints everywhere
diff --git a/.kiro/steering/project-context.md b/.kiro/steering/project-context.md
index ffd814f..0ebb857 100644
--- a/.kiro/steering/project-context.md
+++ b/.kiro/steering/project-context.md
@@ -40,14 +40,17 @@ Three-layer signal aggregation engine:
- Container registry: `registry.celestium.life/stonks-oracle`
## CI/CD
-- GitHub Actions workflow at `.github/workflows/build.yml`
-- Push to `main` triggers: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
+- Woodpecker CI pipelines in `.woodpecker/` — triggered by push to `main` on Gitea
+- Push to Gitea: `git push gitea main` — this is the primary push target
+- ArgoCD watches Gitea `main` and auto-syncs beta/paper/live stages
+- Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
- Images tagged as `registry.celestium.life/stonks-oracle/:` and `:latest`
- Dashboard image: `frontend/Dockerfile` (multi-stage: node:24 → nginx-unprivileged on port 8080)
- Superset image: `docker/Dockerfile.superset` (apache/superset + trino + psycopg2)
- Python service images: `docker/Dockerfile` with `SERVICE_CMD` build arg
- Let CI handle image builds and pushes — do NOT manually `docker build && docker push`
-- Check CI status: `gh run list -L 3`
+- **Do NOT push directly to GitHub** — GitHub (`origin`) is the promotion target after CI builds and tests pass
+- Promotion to GitHub: `git push origin main` (only after Woodpecker CI succeeds)
## Deployment Scripts
- `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available)
@@ -76,9 +79,9 @@ When a full reset is needed:
- Ollama: `ollama.ollama-service.svc.cluster.local:11434` (cluster-internal), also at `http://10.1.1.12:2701` (external), GPU: 4070 Ti Super 16GB
## Database Migrations
-- Located in `infra/migrations/001_*.sql` through `027_*.sql`
+- Located in `infra/migrations/001_*.sql` through `030_*.sql`
- Applied automatically by `runmefirst.sh` in sorted order
-- Next migration number: **029**
+- Next migration number: **031**
- Key migrations:
- 016: Global news interpolation (global_events, macro_impact_records, exposure_profiles, trend_projections)
- 017: Competitive intelligence (competitor_relationships, competitive_signal_records)
diff --git a/README.md b/README.md
index 7dbb3c5..c540a08 100644
--- a/README.md
+++ b/README.md
@@ -8,6 +8,21 @@ Licensed under the [Business Source License 1.1](LICENSE). Production use requir
AI-powered market intelligence and autonomous paper-trading platform. Ingests market data, company news, and regulatory filings; extracts structured intelligence with local LLMs; aggregates signals across three layers (company, macro, competitive); and autonomously executes paper trades — all self-hosted on Kubernetes.
+## Documentation
+
+| Document | Description |
+|----------|-------------|
+| [Service Reference](docs/services.md) | All 13 services — purpose, configuration, queue topology, database tables |
+| [API Reference](docs/api-reference.md) | Complete endpoint reference for Query API, Symbol Registry, Trading, and Risk services |
+| [Helm Chart Reference](docs/helm-reference.md) | All Helm values: services, config, secrets, ingress, network policies, analytics stack |
+| [Docker Deployment Guide](docs/docker-deployment.md) | Docker Compose setup, environment variables, volumes, operational commands |
+| [Kubernetes Architecture](docs/architecture-kubernetes.md) | Mermaid diagram of the K8s deployment topology, namespaces, ingress, and secrets |
+| [Docker Compose Architecture](docs/architecture-docker-compose.md) | Mermaid diagram of all containers, port mappings, volumes, and dependencies |
+| [Data Pipeline Architecture](docs/architecture-data-pipeline.md) | Mermaid diagram of the end-to-end data pipeline, queue topology, and signal layers |
+| [AI Agents Guide](docs/ai-agents.md) | Built-in agents, variant management, prompt tuning, and performance monitoring |
+| [Backup & Restore Guide](docs/backup-restore.md) | Backup scripts, restore procedures, retention policies, and disaster recovery |
+| [Observability Reference](docs/observability.md) | Prometheus metrics, alerting rules, structured logging, and dead-letter queues |
+
## What It Does
Stonks Oracle tracks 50 companies across 10 sectors. It monitors multiple data sources, runs every article and filing through a local Ollama model to extract structured intelligence, aggregates those signals into rolling trend summaries with contradiction detection, and generates explainable trade recommendations. An autonomous trading engine then evaluates those recommendations and executes paper trades through Alpaca without manual intervention.
@@ -16,41 +31,47 @@ Everything is auditable — raw artifacts, prompts, model outputs, decision trac
## Architecture
+```mermaid
+flowchart LR
+ subgraph sources ["Data Sources"]
+ polygon["Polygon.io"]
+ sec["SEC EDGAR"]
+ macro_src["Macro News"]
+ end
+
+ subgraph pipeline ["Signal Processing"]
+ scheduler["Scheduler"]
+ ingestion["Ingestion"]
+ parser["Parser"]
+ extractor["Extractor"]
+ aggregation["Aggregation"]
+ recommendation["Recommendation"]
+ end
+
+ subgraph trading ["Trading"]
+ risk["Risk Engine"]
+ engine["Trading Engine"]
+ broker["Broker Adapter"]
+ alpaca["Alpaca (paper)"]
+ end
+
+ subgraph analytics ["Analytics"]
+ lake["Lake Publisher"]
+ trino["Trino"]
+ superset["Superset"]
+ dashboard["Dashboard"]
+ end
+
+ sources --> scheduler --> ingestion --> parser --> extractor --> aggregation --> recommendation
+ recommendation --> risk --> engine --> broker --> alpaca
+ aggregation --> lake --> trino --> superset
+ trino --> dashboard
```
- ┌──────────────────────────────────────────┐
- │ Signal Aggregation │
- │ │
-┌───────────┐ ┌──────────┐ │ ┌──────────┐ ┌────────────────┐ │
-│ Scheduler │─▶│Ingestion │─▶│ │ Parser │─▶│ Extractor │ │
-└───────────┘ └──────────┘ │ └──────────┘ └──────┬─────────┘ │
- │ │ │
- │ ┌─────────────┘ │
- │ ▼ │
- │ ┌─────────────┐ ┌────────────────┐ │
- │ │ Aggregation │───▶│ Recommendation │ │
- │ └──────┬──────┘ └───────┬────────┘ │
- │ │ │ │
- │ Macro signals Competitive │
- │ + Competitive signals │
- │ signals merged │
- └──────────────────────────────────────────┘
- │
- ┌───────────────────────┘
- ▼
- ┌─────────────┐ ┌────────────────┐ ┌──────────────┐
- │ Risk Engine │───▶│ Trading Engine │───▶│Broker Adapter│
- └─────────────┘ └────────────────┘ └──────────────┘
- │
- ┌────────────────┐ ┌──────────┘
- │ Lake Publisher │ ▼
- └───────┬────────┘ Alpaca (paper)
- │
- ┌──────────────┼──────────────┐
- ▼ ▼ ▼
- ┌──────────┐ ┌──────────┐ ┌───────────┐
- │ Trino │ │ Superset │ │ Dashboard │
- └──────────┘ └──────────┘ └───────────┘
-```
+
+For detailed architecture diagrams see:
+- [Kubernetes Deployment](docs/architecture-kubernetes.md)
+- [Docker Compose Deployment](docs/architecture-docker-compose.md)
+- [Data Pipeline](docs/architecture-data-pipeline.md)
Two planes:
- **Operational** — ingestion, parsing, extraction, aggregation, recommendations, risk evaluation, autonomous trading, trade execution (PostgreSQL, Redis, MinIO)
diff --git a/docker-compose.yml b/docker-compose.yml
index 31ef001..944d79b 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,6 +1,21 @@
version: "3.9"
+x-app-env: &app-env
+ POSTGRES_HOST: postgres
+ POSTGRES_PORT: "5432"
+ POSTGRES_DB: stonks
+ POSTGRES_USER: stonks
+ POSTGRES_PASSWORD: stonks_dev
+ REDIS_HOST: redis
+ REDIS_PORT: "6379"
+ MINIO_ENDPOINT: minio:9000
+ MINIO_ACCESS_KEY: minioadmin
+ MINIO_SECRET_KEY: minioadmin
+ OLLAMA_BASE_URL: http://ollama:11434
+
services:
+ # ── Infrastructure ──────────────────────────────────────────────
+
postgres:
image: postgres:16-alpine
environment:
@@ -109,6 +124,291 @@ services:
depends_on:
- trino
+ # ── Application Services ────────────────────────────────────────
+
+ scheduler:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile.scheduler
+ environment:
+ <<: *app-env
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "pgrep -f 'python -m services.scheduler.app' || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ symbol-registry:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000"
+ environment:
+ <<: *app-env
+ ports:
+ - "8001:8000"
+ depends_on:
+ postgres:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ ingestion:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "python -m services.ingestion.worker"
+ environment:
+ <<: *app-env
+ env_file:
+ - .env
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ minio:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "pgrep -f 'python -m services.ingestion.worker' || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ parser:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "python -m services.parser.worker"
+ environment:
+ <<: *app-env
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "pgrep -f 'python -m services.parser.worker' || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ extractor:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "python -m services.extractor.main"
+ environment:
+ <<: *app-env
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ ollama:
+ condition: service_started
+ healthcheck:
+ test: ["CMD-SHELL", "pgrep -f 'python -m services.extractor.main' || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ aggregation:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "python -m services.aggregation.main"
+ environment:
+ <<: *app-env
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "pgrep -f 'python -m services.aggregation.main' || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ recommendation:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "python -m services.recommendation.main"
+ environment:
+ <<: *app-env
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "pgrep -f 'python -m services.recommendation.main' || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ trading-engine:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "uvicorn services.trading.app:app --host 0.0.0.0 --port 8000"
+ environment:
+ <<: *app-env
+ env_file:
+ - .env
+ ports:
+ - "8002:8000"
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ risk-engine:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "uvicorn services.risk.app:app --host 0.0.0.0 --port 8000"
+ environment:
+ <<: *app-env
+ ports:
+ - "8003:8000"
+ depends_on:
+ postgres:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ broker-adapter:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "python -m services.adapters.broker_service"
+ environment:
+ <<: *app-env
+ env_file:
+ - .env
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "pgrep -f 'python -m services.adapters.broker_service' || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ lake-publisher:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "python -m services.lake_publisher.jobs"
+ environment:
+ <<: *app-env
+ depends_on:
+ postgres:
+ condition: service_healthy
+ minio:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "pgrep -f 'python -m services.lake_publisher.jobs' || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ query-api:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "uvicorn services.api.app:app --host 0.0.0.0 --port 8000"
+ environment:
+ <<: *app-env
+ ports:
+ - "8004:8000"
+ depends_on:
+ postgres:
+ condition: service_healthy
+ redis:
+ condition: service_healthy
+ minio:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 15s
+ restart: unless-stopped
+
+ dashboard:
+ build:
+ context: ./frontend
+ dockerfile: Dockerfile
+ ports:
+ - "3000:8080"
+ depends_on:
+ query-api:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "curl -f http://localhost:8080/ || exit 1"]
+ interval: 10s
+ timeout: 5s
+ retries: 3
+ start_period: 10s
+ restart: unless-stopped
+
volumes:
pgdata:
miniodata:
diff --git a/docs/ai-agents.md b/docs/ai-agents.md
new file mode 100644
index 0000000..58914c1
--- /dev/null
+++ b/docs/ai-agents.md
@@ -0,0 +1,618 @@
+# AI Agent Building Guide
+
+Stonks Oracle uses three AI agents powered by a local Ollama instance. Each agent has a dedicated purpose in the pipeline, a database-backed configuration, and support for A/B testing through variants. This guide covers how each agent works, how to configure them, how to create and test variants, and how to monitor performance.
+
+## Table of Contents
+
+- [Built-in Agents](#built-in-agents)
+ - [Document Intelligence Extractor](#1-document-intelligence-extractor)
+ - [Global Event Classifier](#2-global-event-classifier)
+ - [Thesis Rewriter](#3-thesis-rewriter)
+- [Database Schema](#database-schema)
+ - [ai_agents Table](#ai_agents-table)
+ - [agent_variants Table](#agent_variants-table)
+ - [agent_performance_log Table](#agent_performance_log-table)
+- [AgentConfigResolver](#agentconfigresolver)
+- [Performance Logging and Variant Comparison](#performance-logging-and-variant-comparison)
+- [API Endpoints](#api-endpoints)
+- [Step-by-Step: Creating and Activating a Variant](#step-by-step-creating-and-activating-a-variant)
+
+---
+
+## Built-in Agents
+
+Three agents are seeded into the `ai_agents` table on first migration (migration `026_ai_agents.sql`). They have `source = 'system'` and cannot be deleted through the API — only deactivated or edited.
+
+### 1. Document Intelligence Extractor
+
+| Field | Value |
+|-------|-------|
+| **Slug** | `document-extractor` |
+| **Purpose** | Extracts structured intelligence (sentiment, catalysts, impact scores, key facts, risks) from company news, SEC filings, earnings transcripts, and press releases |
+| **Default Model** | `qwen3.5:9b-fast` (Ollama) |
+| **Prompt Version** | `document-intel-v2` |
+| **Schema Version** | `2.0.0` |
+| **Entry Point** | `services/extractor/main.py` → `services/extractor/client.py` |
+
+**Input Data:**
+- Normalized document text (fetched from MinIO or passed in the Redis job payload)
+- Document type: `article`, `filing`, `transcript`, or `press_release`
+- List of tracked tickers for company identification
+- Document ID for traceability
+
+**Output Schema** (`ExtractionResult`):
+
+```json
+{
+ "summary": "1-3 sentence summary",
+ "companies": [
+ {
+ "ticker": "AAPL",
+ "company_name": "Apple Inc.",
+ "relevance": 0.9,
+ "sentiment": "positive|negative|neutral|mixed",
+ "impact_score": 0.7,
+ "impact_horizon": "intraday|1d|1d_7d|1d_30d|30d_90d|90d_plus",
+ "catalyst_type": "earnings|product|legal|macro|supply_chain|m_and_a|rating_change|other",
+ "key_facts": ["fact1", "fact2"],
+ "risks": ["risk1"],
+ "evidence_spans": ["verbatim quote from document"]
+ }
+ ],
+ "macro_themes": ["inflation", "ai_capex"],
+ "novelty_score": 0.6,
+ "confidence": 0.8,
+ "extraction_warnings": []
+}
+```
+
+**System Prompt:**
+
+```
+You are a financial document analyst. Extract structured data as JSON.
+Return ONLY a single JSON object. No markdown fences, no explanation,
+no text before or after the JSON. Every field in the schema is required.
+Use "other" for catalyst_type if unsure. Keep evidence_spans short
+(under 20 words each). Keep key_facts to 3-5 items max.
+```
+
+**User Prompt Template** (built by `build_extraction_prompt()` in `services/extractor/prompts.py`):
+- Includes document type and type-specific guidance (article, filing, transcript, press release)
+- Includes tracked ticker list with rules for company identification
+- Includes the full JSON schema field descriptions
+- Truncates documents to 8,000 characters to limit inference time
+
+---
+
+### 2. Global Event Classifier
+
+| Field | Value |
+|-------|-------|
+| **Slug** | `event-classifier` |
+| **Purpose** | Classifies global/geopolitical news into structured macro events with impact type, severity, affected regions/sectors/commodities, and estimated duration |
+| **Default Model** | `qwen3.5:9b-fast` (Ollama) |
+| **Prompt Version** | `event-classification-v1` |
+| **Schema Version** | `1.0.0` |
+| **Entry Point** | `services/extractor/main.py` → `services/extractor/event_classifier.py` |
+
+**Input Data:**
+- Normalized text of a macro news article (from the `stonks:queue:macro_classification` Redis queue)
+- Document ID for traceability
+
+**Output Schema** (`GlobalEvent`):
+
+```json
+{
+ "event_types": ["trade_barrier", "commodity_shock"],
+ "severity": "low|moderate|high|critical",
+ "affected_regions": ["US", "CN"],
+ "affected_sectors": ["Energy", "Industrials"],
+ "affected_commodities": ["crude_oil"],
+ "summary": "1-3 sentence summary of event and market implications",
+ "key_facts": ["fact1", "fact2"],
+ "estimated_duration": "short_term|medium_term|long_term",
+ "confidence": 0.75
+}
+```
+
+Valid `event_types`: `supply_disruption`, `demand_shift`, `cost_increase`, `regulatory_pressure`, `currency_impact`, `commodity_shock`, `trade_barrier`, `geopolitical_risk`
+
+Valid `severity`: `low`, `moderate`, `high`, `critical`
+
+**System Prompt:**
+
+```
+You classify MACRO-LEVEL global news into structured event JSON.
+Return ONLY a single JSON object. No markdown, no explanation.
+Every field is required. Keep key_facts to 3-5 items. Keep summary
+under 3 sentences.
+
+CRITICAL: Only classify articles about MACRO events that affect entire
+markets, sectors, or economies. Examples: trade wars, interest rate
+changes, commodity supply disruptions, regulatory changes, geopolitical
+conflicts, natural disasters.
+
+DO NOT classify as macro events: individual company earnings, lawsuits
+against a single company, single-company management changes, individual
+stock analysis, company-specific debt or bankruptcy, product launches
+by one company. For these, set severity to "low", confidence below 0.3,
+and leave affected_regions, affected_sectors, and affected_commodities
+as empty arrays.
+```
+
+**User Prompt Template** (built by `build_event_classification_prompt()` in `services/extractor/event_classifier.py`):
+- Includes anti-hallucination rules
+- Lists all valid enum values for each field
+- Truncates articles to 6,000 characters
+
+---
+
+### 3. Thesis Rewriter
+
+| Field | Value |
+|-------|-------|
+| **Slug** | `thesis-rewriter` |
+| **Purpose** | Rewrites deterministic trade thesis summaries into clear, professional analyst prose. Optional layer — the system falls back to the deterministic thesis if this fails |
+| **Default Model** | `qwen3.5:9b-fast` (Ollama) |
+| **Prompt Version** | `thesis-rewrite-v1` |
+| **Schema Version** | `1.0.0` |
+| **Entry Point** | `services/recommendation/main.py` → `services/recommendation/thesis_llm.py` |
+
+**Input Data:**
+- Deterministic thesis string (rule-based, built from trend data and eligibility rules)
+- `TrendSummary` context: ticker, window, direction, strength, confidence, contradiction score, dominant catalysts, material risks
+
+**Output Schema:**
+- Plain text (not JSON). The model returns only the rewritten thesis as a string, under 150 words.
+- On failure or empty response, the original deterministic thesis is returned unchanged.
+
+**System Prompt:**
+
+```
+You are a concise financial analyst. You rewrite structured trade thesis
+summaries into clear, professional prose suitable for an internal
+research note.
+
+STRICT RULES:
+1. Do NOT add any information that is not present in the input.
+2. Do NOT fabricate numbers, dates, company names, or analyst opinions.
+3. Keep the rewrite under 150 words.
+4. Preserve all factual claims, risk notes, and evidence counts from
+ the input.
+5. Use a neutral, professional tone. Avoid hype or marketing language.
+6. Return ONLY the rewritten thesis text. No JSON, no markdown, no
+ commentary.
+```
+
+**User Prompt Template** (built by `build_thesis_rewrite_prompt()` in `services/recommendation/thesis_llm.py`):
+- Includes the deterministic thesis between delimiters
+- Includes trend context: ticker, window, direction, strength, confidence, contradiction score, top catalysts, top risks
+
+---
+
+## Database Schema
+
+### `ai_agents` Table
+
+Defined in migration `026_ai_agents.sql`. Stores the base configuration for each agent.
+
+| Column | Type | Default | Description |
+|--------|------|---------|-------------|
+| `id` | `UUID` | `gen_random_uuid()` | Primary key |
+| `name` | `VARCHAR(100)` | — | Human-readable name (unique) |
+| `slug` | `VARCHAR(100)` | — | URL-safe identifier (unique), used by `AgentConfigResolver` |
+| `purpose` | `TEXT` | `''` | Description of what the agent does |
+| `model_provider` | `VARCHAR(50)` | `'ollama'` | LLM provider |
+| `model_name` | `VARCHAR(200)` | `'qwen3.5:9b'` | Model identifier |
+| `system_prompt` | `TEXT` | `''` | System prompt sent to the model |
+| `user_prompt_template` | `TEXT` | `''` | User prompt template (optional — code-defined templates take precedence) |
+| `prompt_version` | `VARCHAR(100)` | `''` | Version tag for prompt tracking |
+| `schema_version` | `VARCHAR(50)` | `'1.0.0'` | Version of the output schema |
+| `temperature` | `FLOAT` | `0.0` | Model temperature |
+| `max_tokens` | `INTEGER` | `32768` | Maximum output tokens |
+| `timeout_seconds` | `INTEGER` | `120` | Request timeout |
+| `max_retries` | `INTEGER` | `2` | Retry count on failure |
+| `active` | `BOOLEAN` | `TRUE` | Whether the agent is enabled |
+| `source` | `VARCHAR(20)` | `'system'` | `'system'` for built-in agents, `'user'` for API-created |
+| `created_at` | `TIMESTAMPTZ` | `NOW()` | Creation timestamp |
+| `updated_at` | `TIMESTAMPTZ` | `NOW()` | Last update timestamp |
+
+**Indexes:**
+- `idx_ai_agents_slug` on `slug`
+- `idx_ai_agents_active` on `active`
+
+**Registration:**
+- **System-seeded**: The three built-in agents are inserted by migration 026 using `INSERT ... WHERE NOT EXISTS` — they are only created if no row with that slug exists. This means user edits to system agents are preserved across re-migrations.
+- **API-created**: Users can create custom agents via `POST /api/agents`. These get `source = 'user'` and can be deleted.
+
+### `agent_variants` Table
+
+Defined in migration `027_agent_variants.sql`. Stores alternative configurations for A/B testing.
+
+| Column | Type | Default | Description |
+|--------|------|---------|-------------|
+| `id` | `UUID` | `gen_random_uuid()` | Primary key |
+| `agent_id` | `UUID` | — | Foreign key → `ai_agents(id)` (CASCADE delete) |
+| `variant_name` | `VARCHAR(200)` | — | Human-readable variant name |
+| `variant_slug` | `VARCHAR(200)` | — | URL-safe slug (unique per agent) |
+| `description` | `TEXT` | `''` | What this variant changes |
+| `model_provider` | `VARCHAR(50)` | `'ollama'` | LLM provider override |
+| `model_name` | `VARCHAR(200)` | — | Model override |
+| `system_prompt` | `TEXT` | `''` | System prompt override |
+| `user_prompt_template` | `TEXT` | `''` | User prompt template override |
+| `prompt_version` | `VARCHAR(100)` | `''` | Prompt version tag |
+| `temperature` | `FLOAT` | `0.0` | Temperature override |
+| `max_tokens` | `INTEGER` | `32768` | Max tokens override |
+| `context_window` | `INTEGER` | `0` | Ollama `num_ctx` override (0 = model default) |
+| `input_token_limit` | `INTEGER` | `0` | Max input tokens before truncation (0 = no limit) |
+| `token_budget` | `INTEGER` | `0` | Total tokens per hour budget (0 = unlimited) |
+| `timeout_seconds` | `INTEGER` | `120` | Timeout override |
+| `max_retries` | `INTEGER` | `2` | Retry count override |
+| `is_active` | `BOOLEAN` | `FALSE` | Whether this variant is the active override |
+| `created_at` | `TIMESTAMPTZ` | `NOW()` | Creation timestamp |
+| `updated_at` | `TIMESTAMPTZ` | `NOW()` | Last update timestamp |
+
+**Indexes and Constraints:**
+- `idx_agent_variants_slug` — unique index on `(agent_id, variant_slug)` — each agent's variant slugs must be unique
+- `idx_agent_variants_active` — unique partial index on `(agent_id) WHERE is_active = TRUE` — **at most one active variant per agent** (database-enforced)
+- `idx_agent_variants_agent` — lookup by agent
+
+### `agent_performance_log` Table
+
+Defined in migration `026_ai_agents.sql`, extended in `027_agent_variants.sql` with `variant_id`.
+
+| Column | Type | Default | Description |
+|--------|------|---------|-------------|
+| `id` | `UUID` | `gen_random_uuid()` | Primary key |
+| `agent_id` | `UUID` | — | Foreign key → `ai_agents(id)` (CASCADE delete) |
+| `variant_id` | `UUID` | `NULL` | Foreign key → `agent_variants(id)` (SET NULL on delete) |
+| `document_id` | `UUID` | `NULL` | Foreign key → `documents(id)` (SET NULL on delete) |
+| `ticker` | `VARCHAR(20)` | — | Stock ticker processed |
+| `success` | `BOOLEAN` | — | Whether the invocation succeeded |
+| `duration_ms` | `INTEGER` | `0` | Total invocation time in milliseconds |
+| `confidence` | `FLOAT` | `0.0` | Model confidence score (0.0 for thesis rewrites) |
+| `retry_count` | `INTEGER` | `0` | Number of retries before success/failure |
+| `input_tokens` | `INTEGER` | `0` | Estimated input tokens (chars / 4) |
+| `output_tokens` | `INTEGER` | `0` | Estimated output tokens (chars / 4) |
+| `error_message` | `TEXT` | `NULL` | Error description on failure |
+| `recorded_at` | `TIMESTAMPTZ` | `NOW()` | When the invocation occurred |
+
+**Indexes:**
+- `idx_agent_perf_agent` on `(agent_id, recorded_at DESC)`
+- `idx_agent_perf_time` on `(recorded_at DESC)`
+- `idx_agent_perf_variant` on `(variant_id, recorded_at DESC)`
+
+---
+
+## AgentConfigResolver
+
+**Module:** `services/shared/agent_config.py`
+
+The `AgentConfigResolver` is the central mechanism for resolving runtime agent configuration. All three agent services use it instead of duplicating resolution logic.
+
+### How It Works
+
+1. **Lookup by slug**: The resolver queries the `ai_agents` table by slug (e.g., `"document-extractor"`), joining with `agent_variants` to find any active variant.
+
+2. **COALESCE-based override**: The SQL query uses `COALESCE(variant_column, agent_column)` for every configuration field. If an active variant exists and has a non-NULL value for a field, that value is used. Otherwise, the base agent's value is used.
+
+ ```sql
+ SELECT a.id AS agent_id,
+ v.id AS variant_id,
+ COALESCE(v.model_provider, a.model_provider) AS model_provider,
+ COALESCE(v.model_name, a.model_name) AS model_name,
+ COALESCE(v.system_prompt, a.system_prompt) AS system_prompt,
+ COALESCE(v.user_prompt_template, a.user_prompt_template) AS user_prompt_template,
+ -- ... all other fields ...
+ FROM ai_agents a
+ LEFT JOIN agent_variants v
+ ON v.agent_id = a.id AND v.is_active = TRUE
+ WHERE a.slug = $1
+ AND a.active = TRUE
+ ```
+
+3. **TTL cache (60 seconds)**: Resolved configurations are cached in memory using `time.monotonic()`. Cache entries expire after 60 seconds (configurable via `ttl_seconds`). This means variant swaps take effect within 60 seconds without restarting any service.
+
+4. **Fallback behavior**: If the database query fails or returns no rows (agent not found or inactive), the resolver returns `None`. Callers fall back to environment-variable-based `OllamaConfig` defaults.
+
+### Resolved Config Dataclass
+
+```python
+@dataclass(frozen=True, slots=True)
+class ResolvedAgentConfig:
+ agent_id: str
+ variant_id: str | None # None if no active variant
+ model_provider: str
+ model_name: str
+ system_prompt: str
+ user_prompt_template: str
+ prompt_version: str
+ temperature: float
+ max_tokens: int
+ context_window: int # Ollama num_ctx; 0 = model default
+ input_token_limit: int # Max input chars before truncation; 0 = no limit
+ token_budget: int # Hourly token budget; 0 = unlimited
+ timeout_seconds: int
+ max_retries: int
+```
+
+### Usage Pattern
+
+```python
+from services.shared.agent_config import AgentConfigResolver
+
+resolver = AgentConfigResolver(pool, ttl_seconds=60)
+config = await resolver.resolve("document-extractor")
+
+if config is None:
+ # Fall back to env-var defaults
+ ...
+else:
+ # Use config.model_name, config.system_prompt, etc.
+ ...
+```
+
+### Cache Invalidation
+
+```python
+resolver.invalidate("document-extractor") # Clear one entry
+resolver.invalidate() # Clear all entries
+```
+
+### Config Refresh in Workers
+
+The extractor and recommendation workers periodically re-resolve their agent config (every 100 jobs for the extractor, every 50 jobs for the recommendation worker). If the resolved model changes, the worker creates a new `OllamaClient` instance with the updated configuration.
+
+---
+
+## Performance Logging and Variant Comparison
+
+Every agent invocation is logged to `agent_performance_log` with the `agent_id` and `variant_id` (if a variant was active). This enables comparing variant effectiveness.
+
+### What Gets Logged
+
+- **Document extractor**: Logged in `services/extractor/main.py` after each extraction. Records success/failure, duration, confidence, retry count, token estimates.
+- **Event classifier**: Logged in `services/extractor/event_classifier.py` after each classification. Same fields.
+- **Thesis rewriter**: Logged in `services/recommendation/thesis_llm.py` after each rewrite attempt. Confidence is always 0.0 (not applicable for rewrites).
+
+### Querying for Variant Comparison
+
+Compare two variants of the document extractor over the last 24 hours:
+
+```sql
+SELECT
+ v.variant_name,
+ COUNT(*) AS total_invocations,
+ COUNT(*) FILTER (WHERE p.success) AS successes,
+ ROUND(100.0 * COUNT(*) FILTER (WHERE p.success) / COUNT(*), 1) AS success_rate_pct,
+ ROUND(AVG(p.duration_ms)::numeric) AS avg_duration_ms,
+ ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY p.duration_ms)::numeric) AS p95_duration_ms,
+ ROUND(AVG(p.confidence)::numeric, 4) AS avg_confidence,
+ ROUND(AVG(p.retry_count)::numeric, 2) AS avg_retries,
+ SUM(p.input_tokens + p.output_tokens) AS total_tokens
+FROM agent_performance_log p
+JOIN agent_variants v ON v.id = p.variant_id
+WHERE p.agent_id = ''
+ AND p.recorded_at >= NOW() - INTERVAL '24 hours'
+GROUP BY v.variant_name
+ORDER BY success_rate_pct DESC;
+```
+
+Compare base agent (no variant) vs active variant:
+
+```sql
+SELECT
+ CASE WHEN p.variant_id IS NULL THEN 'base' ELSE v.variant_name END AS config,
+ COUNT(*) AS invocations,
+ ROUND(100.0 * COUNT(*) FILTER (WHERE p.success) / COUNT(*), 1) AS success_rate_pct,
+ ROUND(AVG(p.duration_ms)::numeric) AS avg_duration_ms,
+ ROUND(AVG(p.confidence)::numeric, 4) AS avg_confidence
+FROM agent_performance_log p
+LEFT JOIN agent_variants v ON v.id = p.variant_id
+WHERE p.agent_id = ''
+ AND p.recorded_at >= NOW() - INTERVAL '48 hours'
+GROUP BY config
+ORDER BY config;
+```
+
+### Token Budget Enforcement
+
+Variants can set a `token_budget` (total tokens per hour). Before each invocation, the worker checks:
+
+```sql
+SELECT COALESCE(SUM(input_tokens + output_tokens), 0) AS total_tokens
+FROM agent_performance_log
+WHERE variant_id = $1
+ AND recorded_at >= NOW() - INTERVAL '1 hour'
+```
+
+If the budget is exceeded, the invocation is skipped (extractor) or falls back to the deterministic thesis (thesis rewriter).
+
+---
+
+## API Endpoints
+
+All agent endpoints are served by the Query API (`services/api/app.py`) under the `/api/agents` prefix.
+
+### Agent CRUD
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/agents` | List all agents. Query param: `active_only` (bool, default `false`) |
+| `GET` | `/api/agents/{agent_id}` | Get a single agent by UUID |
+| `POST` | `/api/agents` | Create a new user-defined agent (returns 201) |
+| `PUT` | `/api/agents/{agent_id}` | Partial update an agent (system or user) |
+| `DELETE` | `/api/agents/{agent_id}` | Delete a user-created agent. Returns 403 for system agents |
+
+**Create Agent Request Body:**
+
+```json
+{
+ "name": "My Custom Agent",
+ "slug": "my-custom-agent",
+ "purpose": "Custom extraction for earnings calls",
+ "model_provider": "ollama",
+ "model_name": "llama3.1:8b",
+ "system_prompt": "You are a financial analyst...",
+ "user_prompt_template": "",
+ "prompt_version": "v1",
+ "schema_version": "1.0.0",
+ "temperature": 0.0,
+ "max_tokens": 32768,
+ "timeout_seconds": 120,
+ "max_retries": 2
+}
+```
+
+**Update Agent Request Body** (all fields optional):
+
+```json
+{
+ "model_name": "qwen3.5:14b",
+ "system_prompt": "Updated prompt...",
+ "temperature": 0.1,
+ "active": false
+}
+```
+
+### Agent Performance
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/agents/{agent_id}/performance` | Aggregated metrics. Query param: `hours` (int, default 24, max 720) |
+| `GET` | `/api/agents/{agent_id}/performance/history` | Hourly time-series. Query param: `hours` (int, default 24, max 720) |
+
+**Performance Response:**
+
+```json
+{
+ "total_invocations": 1250,
+ "successes": 1180,
+ "failures": 70,
+ "avg_duration_ms": 3400,
+ "p95_duration_ms": 8200,
+ "avg_confidence": 0.7234,
+ "avg_retries": 0.15,
+ "total_input_tokens": 5000000,
+ "total_output_tokens": 1200000,
+ "success_rate": 0.944
+}
+```
+
+### Variant CRUD
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/agents/{agent_id}/variants` | List all variants for an agent |
+| `GET` | `/api/agents/{agent_id}/variants/{variant_id}` | Get a single variant |
+| `POST` | `/api/agents/{agent_id}/variants` | Create a new variant (returns 201, 409 on duplicate slug) |
+| `PUT` | `/api/agents/{agent_id}/variants/{variant_id}` | Partial update a variant |
+| `DELETE` | `/api/agents/{agent_id}/variants/{variant_id}` | Delete a variant (returns 400 if active) |
+
+### Clone Endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `POST` | `/api/agents/{agent_id}/clone` | Clone an agent's base config as a new variant |
+| `POST` | `/api/agents/{agent_id}/variants/{variant_id}/clone` | Clone an existing variant as a new variant |
+
+Clone requests copy all configuration fields from the source, with optional overrides in the request body.
+
+### Activate / Deactivate
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `POST` | `/api/agents/{agent_id}/variants/{variant_id}/activate` | Set a variant as active (deactivates any other active variant in a single transaction) |
+| `POST` | `/api/agents/{agent_id}/variants/deactivate` | Deactivate the currently active variant (agent falls back to base config) |
+
+### Per-Variant Performance
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/agents/{agent_id}/variants/{variant_id}/performance` | Aggregated metrics for a specific variant |
+| `GET` | `/api/agents/{agent_id}/variants/{variant_id}/performance/history` | Hourly time-series for a specific variant |
+
+---
+
+## Step-by-Step: Creating and Activating a Variant
+
+This walkthrough creates a new variant of the document extractor that uses a different model and activates it for live traffic.
+
+### 1. Find the Agent ID
+
+```bash
+curl -s https://stonks-api.celestium.life/api/agents?active_only=true | jq '.[] | select(.slug == "document-extractor") | .id'
+```
+
+Note the UUID — we'll call it `AGENT_ID`.
+
+### 2. Clone the Agent as a Variant
+
+```bash
+curl -s -X POST https://stonks-api.celestium.life/api/agents/$AGENT_ID/clone \
+ -H "Content-Type: application/json" \
+ -d '{
+ "variant_name": "Llama 3.1 8B Test",
+ "description": "Testing llama3.1:8b as an alternative to qwen3.5:9b-fast",
+ "model_name": "llama3.1:8b",
+ "temperature": 0.1
+ }' | jq .
+```
+
+This creates a new variant with all fields copied from the base agent, except `model_name` and `temperature` which are overridden. The variant starts as `is_active: false`.
+
+Note the variant's `id` — we'll call it `VARIANT_ID`.
+
+### 3. Activate the Variant
+
+```bash
+curl -s -X POST \
+ https://stonks-api.celestium.life/api/agents/$AGENT_ID/variants/$VARIANT_ID/activate | jq .
+```
+
+This atomically deactivates any previously active variant and activates the new one. Within 60 seconds (the TTL cache window), the extractor worker will pick up the new configuration and start using `llama3.1:8b`.
+
+### 4. Monitor Performance
+
+Wait for some documents to be processed, then compare:
+
+```bash
+# Base agent performance (all invocations)
+curl -s "https://stonks-api.celestium.life/api/agents/$AGENT_ID/performance?hours=4" | jq .
+
+# Variant-specific performance
+curl -s "https://stonks-api.celestium.life/api/agents/$AGENT_ID/variants/$VARIANT_ID/performance?hours=4" | jq .
+```
+
+Check the hourly trend:
+
+```bash
+curl -s "https://stonks-api.celestium.life/api/agents/$AGENT_ID/variants/$VARIANT_ID/performance/history?hours=12" | jq .
+```
+
+### 5. Roll Back (Deactivate)
+
+If the variant underperforms, deactivate it to revert to the base agent config:
+
+```bash
+curl -s -X POST \
+ https://stonks-api.celestium.life/api/agents/$AGENT_ID/variants/deactivate | jq .
+```
+
+The extractor will revert to the base `qwen3.5:9b-fast` configuration within 60 seconds.
+
+### 6. Iterate
+
+You can update the variant's prompt or parameters without creating a new one:
+
+```bash
+curl -s -X PUT \
+ https://stonks-api.celestium.life/api/agents/$AGENT_ID/variants/$VARIANT_ID \
+ -H "Content-Type: application/json" \
+ -d '{
+ "system_prompt": "You are a financial document analyst. Extract structured data as JSON. Be extra conservative with impact scores — only assign > 0.7 for material events with concrete numbers.",
+ "prompt_version": "document-intel-v2-conservative"
+ }' | jq .
+```
+
+Then re-activate and compare again.
diff --git a/docs/api-reference.md b/docs/api-reference.md
new file mode 100644
index 0000000..c460447
--- /dev/null
+++ b/docs/api-reference.md
@@ -0,0 +1,1141 @@
+# Stonks Oracle — API Reference
+
+This document covers every HTTP endpoint exposed by the four FastAPI services in the Stonks Oracle platform. For each endpoint: HTTP method, path, query parameters (with type, default, and constraints), request body schema, response schema, and error codes.
+
+**Live endpoints:**
+
+| Service | Base URL | Source |
+|---------|----------|--------|
+| Query API | `https://stonks-api.celestium.life` | `services/api/app.py` |
+| Symbol Registry | `https://stonks-registry.celestium.life` | `services/symbol_registry/app.py` |
+| Trading Engine | `https://stonks-trading.celestium.life` | `services/trading/app.py` |
+| Risk Engine | (cluster-internal) | `services/risk/app.py` |
+
+**Common error format:** All services return errors as `{"detail": "error message"}` with the appropriate HTTP status code.
+
+---
+
+## Table of Contents
+
+- [1. Query API](#1-query-api)
+- [2. Symbol Registry API](#2-symbol-registry-api)
+- [3. Trading Engine API](#3-trading-engine-api)
+- [4. Risk Engine API](#4-risk-engine-api)
+
+---
+
+## 1. Query API
+
+Source: `services/api/app.py`
+Base path: `/` (most endpoints prefixed with `/api/`)
+
+### 1.1 Health and Metrics
+
+#### `GET /health`
+Liveness probe. Verifies database connectivity.
+
+- **Response:** `{"status": "ok"}`
+- **Errors:** `503` — Database unavailable
+
+#### `GET /metrics`
+Prometheus metrics endpoint for scraping.
+
+- **Response:** Prometheus text format (`text/plain`)
+
+### 1.2 Companies
+
+#### `GET /api/companies`
+List tracked companies with optional filters.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `active` | bool | `true` | Filter by active status |
+| `sector` | string | — | Filter by sector |
+| `ticker` | string | — | Filter by ticker (auto-uppercased) |
+
+- **Response:** Array of company objects with `id`, `ticker`, `legal_name`, `exchange`, `sector`, `industry`, `market_cap_bucket`, `active`, `created_at`, `updated_at`
+
+#### `GET /api/companies/{company_id}`
+Get a single company with aliases and active source count.
+
+- **Path params:** `company_id` (UUID string)
+- **Response:** Company object + `aliases[]` + `active_source_count`
+- **Errors:** `404` — Company not found
+
+#### `GET /api/companies/{company_id}/sources`
+List sources configured for a company.
+
+- **Path params:** `company_id` (UUID string)
+- **Response:** Array of source objects with `id`, `source_type`, `source_name`, `config`, `credibility_score`, `retention_days`, `access_policy`, `active`
+
+### 1.3 Documents
+
+#### `GET /api/documents`
+List documents with optional filters, ordered by `published_at` descending.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `ticker` | string | — | — | Filter by ticker |
+| `company_id` | string | — | — | Filter by company UUID |
+| `document_type` | string | — | — | Filter by type |
+| `status` | string | — | — | Filter by processing status |
+| `since` | string | — | ISO 8601 timestamp | Documents published after this time |
+| `limit` | int | `50` | max `200` | Page size |
+| `offset` | int | `0` | — | Pagination offset |
+
+- **Response:** Array of document objects
+
+#### `GET /api/documents/{document_id}`
+Get a single document with intelligence extraction and company mentions.
+
+- **Path params:** `document_id` (UUID string)
+- **Response:** Document object + `company_mentions[]` + `intelligence` (with `company_impacts[]`)
+- **Errors:** `404` — Document not found
+
+### 1.4 Trends
+
+#### `GET /api/trends`
+List trend summaries with optional filters.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `ticker` | string | — | — | Filter by entity_id (ticker) |
+| `entity_type` | string | `"company"` | — | Entity type filter |
+| `window` | string | — | — | Time window filter |
+| `limit` | int | `50` | max `200` | Page size |
+| `offset` | int | `0` | — | Pagination offset |
+
+- **Response:** Array of trend objects with JSONB fields parsed, plus `projection` sub-object from `trend_projections`
+
+#### `GET /api/trends/history`
+Historical trend snapshots for charting (time series from `trend_history` table).
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `ticker` | string | — | — | Filter by entity_id |
+| `window` | string | — | — | Time window filter |
+| `limit` | int | `200` | max `1000` | Max rows |
+
+- **Response:** Array of trend history objects ordered by `generated_at` ascending
+
+#### `GET /api/trends/{trend_id}`
+Get a single trend summary by ID.
+
+- **Path params:** `trend_id` (UUID string)
+- **Response:** Trend object with parsed JSONB fields
+- **Errors:** `404` — Trend not found
+
+#### `GET /api/trends/{trend_id}/evidence`
+Drill down from a trend window to contributing documents and raw artifacts. Full provenance chain.
+
+- **Path params:** `trend_id` (UUID string)
+- **Response:** `{ trend, evidence[] }` — each evidence item includes `intelligence` and `company_impacts[]`
+- **Errors:** `404` — Trend not found
+
+#### `GET /api/trends/{trend_id}/projection`
+Trend projection for a specific trend window.
+
+- **Path params:** `trend_id` (UUID string)
+- **Response:** Projection object with `projected_direction`, `projected_strength`, `projected_confidence`, `projection_horizon`, `driving_factors`, `macro_contribution_pct`, `diverges_from_current`
+- **Errors:** `404` — Trend not found
+
+### 1.5 Market Prices
+
+#### `GET /api/market/prices/{ticker}`
+Historical close prices from `market_snapshots`.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `limit` | int | `30` | max `200` | Max bars returned |
+
+- **Path params:** `ticker` (auto-uppercased)
+- **Response:** Array of OHLCV objects ordered oldest-first
+
+### 1.6 Recommendations
+
+#### `GET /api/recommendations`
+List recommendations with optional filters.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `ticker` | string | — | — | Filter by ticker |
+| `action` | string | — | — | Filter by action (buy/sell/hold) |
+| `mode` | string | — | — | Filter by mode |
+| `since` | string | — | ISO 8601 | Generated after this time |
+| `min_confidence` | float | — | 0.0–1.0 | Minimum confidence threshold |
+| `limit` | int | `50` | max `200` | Page size |
+| `offset` | int | `0` | — | Pagination offset |
+| `latest` | bool | `true` | — | Return only latest per ticker |
+
+- **Response:** Array of recommendation objects
+
+#### `GET /api/recommendations/{recommendation_id}`
+Get a single recommendation with evidence and risk evaluation.
+
+- **Path params:** `recommendation_id` (UUID string)
+- **Response:** Recommendation + `evidence[]` + `risk_evaluation`
+- **Errors:** `404` — Recommendation not found
+
+#### `GET /api/recommendations/{recommendation_id}/evidence`
+Full evidence drill-down: provenance chain from recommendation to source documents and raw artifacts.
+
+- **Path params:** `recommendation_id` (UUID string)
+- **Response:** `{ recommendation, evidence[], trend_window }` — each evidence item includes `intelligence` and `company_impacts[]`
+- **Errors:** `404` — Recommendation not found
+
+### 1.7 Orders
+
+#### `GET /api/orders`
+List orders with optional filters.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `ticker` | string | — | — | Filter by ticker |
+| `status` | string | — | — | Filter by order status |
+| `side` | string | — | — | Filter by side (buy/sell) |
+| `since` | string | — | ISO 8601 | Created after this time |
+| `limit` | int | `50` | max `200` | Page size |
+| `offset` | int | `0` | — | Pagination offset |
+
+- **Response:** Array of order objects
+
+#### `GET /api/orders/{order_id}`
+Get a single order with events, decision trace, and full audit trail.
+
+- **Path params:** `order_id` (UUID string)
+- **Response:** Order object + `events[]` + `audit_trail`
+- **Errors:** `404` — Order not found
+
+### 1.8 Positions
+
+#### `GET /api/positions`
+List current positions.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `ticker` | string | — | Filter by ticker |
+
+- **Response:** Array of position objects with `id`, `broker_account_id`, `ticker`, `quantity`, `avg_entry_price`, `current_price`, `unrealized_pnl`, `realized_pnl`, `updated_at`
+
+### 1.9 Audit Trail
+
+#### `GET /api/audit/{entity_type}/{entity_id}`
+Get audit events for any entity type and ID.
+
+- **Path params:** `entity_type` (string), `entity_id` (string)
+- **Response:** Array of audit event objects
+- **Errors:** `404` — No audit events found
+
+
+### 1.10 Admin: Source Health
+
+#### `GET /api/admin/sources/health`
+Source health overview with latest ingestion status and failure counts.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `source_type` | string | — | Filter by source type |
+| `company_id` | string | — | Filter by company UUID |
+| `active_only` | bool | `true` | Only show active sources |
+
+- **Response:** Array of source health objects with `source_id`, `source_type`, `source_name`, `credibility_score`, `ticker`, `legal_name`, `last_run_status`, `last_run_at`, `last_error`, `last_items_fetched`, `last_items_new`, `total_runs_24h`, `failed_runs_24h`, `total_items_24h`
+
+#### `GET /api/admin/sources/{source_id}/runs`
+Recent ingestion runs for a specific source.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `limit` | int | `20` | max `100` | Page size |
+| `offset` | int | `0` | — | Pagination offset |
+
+#### `PUT /api/admin/sources/{source_id}/toggle`
+Enable or disable a source.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `active` | bool | `true` | New active state |
+
+- **Errors:** `404` — Source not found
+
+#### `PUT /api/admin/sources/{source_id}/credibility`
+Update a source's credibility score.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `credibility_score` | float | — | 0.0–1.0 | New credibility score |
+
+- **Errors:** `404` — Source not found
+
+### 1.11 Admin: Company Management
+
+#### `PUT /api/admin/companies/{company_id}/toggle`
+Enable or disable a tracked company.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `active` | bool | `true` | New active state |
+
+- **Errors:** `404` — Company not found
+
+#### `PUT /api/admin/companies/{company_id}/sector`
+Update a company's sector and industry classification.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `sector` | string | — | **Required.** New sector |
+| `industry` | string | — | Optional new industry |
+
+- **Errors:** `404` — Company not found
+
+#### `GET /api/admin/companies/coverage`
+Source coverage overview per active company. Shows active source counts by type.
+
+- **Response:** Array of objects with `company_id`, `ticker`, `legal_name`, `sector`, `active_sources`, `market_sources`, `news_sources`, `filings_sources`, `web_scrape_sources`, `broker_sources`
+
+### 1.12 Admin: Trading Configuration
+
+#### `GET /api/admin/trading/config`
+Get the current active risk/trading configuration.
+
+- **Response:** Risk config object with `id`, `name`, `trading_mode`, `config` (JSONB), `active`, timestamps
+
+#### `PUT /api/admin/trading/mode`
+Switch the active trading mode.
+
+| Parameter | Type | Constraints | Description |
+|-----------|------|-------------|-------------|
+| `mode` | string | `paper`, `live`, or `disabled` | **Required.** New trading mode |
+
+#### `PUT /api/admin/trading/config`
+Update the active risk configuration JSON.
+
+- **Body:** `dict[str, Any]` — partial or full risk config object
+- **Response:** Updated config object
+
+#### `GET /api/admin/trading/approvals`
+List pending operator approval requests for live trading orders.
+
+- **Response:** Array of approval objects with `order_job` (JSONB), `recommendation_id`, `ticker`, `side`, `quantity`, `estimated_value`, `status`, `expires_at`, etc.
+
+#### `PUT /api/admin/trading/approvals/{approval_id}`
+Approve or reject a pending operator approval request.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `approved` | bool | — | **Required.** Approve or reject |
+| `reviewed_by` | string | `"operator"` | Reviewer identity |
+| `review_note` | string | `""` | Optional note |
+
+- **Errors:** `404` — Approval not found or no longer pending
+
+#### `GET /api/admin/trading/lockouts`
+List active symbol lockouts (news-shock, cooldown, manual).
+
+#### `POST /api/admin/trading/lockouts`
+Create a manual symbol lockout.
+
+- **Body:** `{ ticker: string, reason: string, duration_minutes: int, lockout_type?: string }`
+- **Errors:** `400` — Missing or invalid fields
+
+#### `DELETE /api/admin/trading/lockouts/{lockout_id}`
+Delete a symbol lockout (early removal).
+
+- **Errors:** `404` — Lockout not found
+
+#### `GET /api/admin/trading/approval-config`
+Get operator approval settings from the active risk config.
+
+- **Response:** `{ auto_approve_paper, require_approval_for_live, approval_timeout_minutes }`
+
+#### `PUT /api/admin/trading/approval-config`
+Update operator approval settings.
+
+- **Body:** `{ auto_approve_paper?: bool, require_approval_for_live?: bool, approval_timeout_minutes?: int }`
+- **Response:** Updated approval settings
+
+
+### 1.13 Operational Dashboard
+
+#### `GET /api/ops/ingestion/throughput`
+Ingestion throughput over time, bucketed by interval.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | 1–168 | Time window |
+| `bucket` | string | `"1h"` | `15m`, `1h`, `6h`, `1d` | Bucket interval |
+
+- **Response:** Array of bucketed throughput objects by source type
+
+#### `GET /api/ops/ingestion/summary`
+High-level ingestion summary for the operational dashboard.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | 1–168 | Time window |
+
+- **Response:** `{ total_runs, completed, failed, pending, running, total_items_fetched, total_items_new, active_sources, active_companies, by_source_type[], hours }`
+
+#### `GET /api/ops/model/failures`
+Recent model extraction failures with error details.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | 1–168 | Time window |
+| `limit` | int | `50` | max `200` | Max results |
+
+#### `GET /api/ops/model/performance`
+Aggregated model performance metrics.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | 1–168 | Time window |
+| `model_name` | string | — | — | Filter by model |
+
+#### `GET /api/ops/pipeline/health`
+Pipeline stage health summary across ingestion, parsing, extraction, and aggregation.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | 1–168 | Time window |
+
+- **Response:** `{ hours, pipeline_enabled, document_stages[], parsing, extraction, aggregation, queue_depths }`
+
+#### `GET /api/ops/pipeline/stream`
+Server-Sent Events stream of live pipeline status. Pushes queue depths and document stage counts every 3 seconds.
+
+- **Response:** `text/event-stream` with JSON data payloads
+
+#### `POST /api/ops/pipeline/retry-failed`
+Re-enqueue documents stuck in `extraction_failed` for another attempt (up to 200).
+
+- **Response:** `{ retried: int, message: string }`
+
+#### `GET /api/ops/pipeline/toggle`
+Get the current pipeline enabled/disabled state.
+
+- **Response:** `{ pipeline_enabled: bool }`
+
+#### `POST /api/ops/pipeline/toggle`
+Toggle the pipeline on or off.
+
+- **Body:** `{ enabled: bool }`
+- **Response:** `{ pipeline_enabled: bool, message: string }`
+
+#### `GET /api/ops/sources/coverage-gaps`
+Identify symbols with missing or insufficient source coverage.
+
+- **Response:** `{ missing_source_types[], stale_sources[] }`
+
+### 1.14 System
+
+#### `GET /api/system/rate-limits`
+Current rate limit configuration and usage.
+
+- **Response:** `{ polygon_global_limit, polygon_source_types, per_type_limits, cadences_seconds, market_api: { rate_per_minute, cadence_seconds, max_tickers_per_cycle, active_sources }, news_api: {...} }`
+
+### 1.15 Analytics
+
+#### `POST /api/analytics/query`
+Proxy SQL to Trino with row limits.
+
+- **Body:** `{ sql: string, limit?: int (default 1000, max 10000) }`
+- **Response:** `{ columns[], rows[][], row_count, elapsed_ms }`
+- **Errors:** `400` — Empty SQL; `502` — Trino connection error; `504` — Trino timeout
+
+#### `GET /api/analytics/schema`
+Trino catalog/schema/table/column metadata for the schema browser.
+
+- **Response:** `{ catalog, schema, tables[{ name, columns[{ name, type }] }] }`
+
+#### `GET /api/analytics/pg-schema`
+PostgreSQL table/column metadata with primary keys, foreign keys, and row estimates.
+
+- **Response:** `{ catalog: "postgresql", schema: "public", tables[] }`
+
+#### `POST /api/analytics/pg-query`
+Run read-only SQL against PostgreSQL directly. Only SELECT statements allowed.
+
+- **Body:** `{ sql: string, limit?: int (default 1000, max 10000) }`
+- **Response:** `{ columns[], rows[][], row_count, elapsed_ms }`
+- **Errors:** `400` — Non-SELECT query, syntax error, table not found, or query error
+
+#### `GET /api/analytics/saved-queries`
+List all saved queries.
+
+#### `POST /api/analytics/saved-queries` (201)
+Save a new query.
+
+- **Body:** `{ name: string, description?: string, sql_text: string }`
+
+#### `DELETE /api/analytics/saved-queries/{query_id}`
+Delete a saved query.
+
+- **Errors:** `404` — Query not found
+
+
+### 1.16 Macro Signal Layer
+
+#### `GET /api/admin/macro/status`
+Return the current macro signal layer enabled/disabled state.
+
+- **Response:** `{ macro_enabled: bool, source: "default" | "risk_configs" }`
+
+#### `PUT /api/admin/macro/toggle`
+Toggle the macro signal layer on or off. Records an audit event.
+
+- **Body:** `{ enabled: bool, operator?: string (default "operator") }`
+- **Response:** `{ macro_enabled, previous_enabled, toggled_by }`
+
+### 1.17 Macro Events and Impacts
+
+#### `GET /api/macro/events`
+List recent global events with filtering.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `severity` | string | — | — | Filter by severity |
+| `region` | string | — | — | Filter by affected region |
+| `sector` | string | — | — | Filter by affected sector |
+| `since` | string | — | ISO 8601 | Events after this time |
+| `until` | string | — | ISO 8601 | Events before this time |
+| `limit` | int | `50` | max `200` | Page size |
+| `offset` | int | `0` | — | Pagination offset |
+
+#### `GET /api/macro/events/{event_id}`
+Event detail with affected companies and macro impact scores.
+
+- **Errors:** `404` — Global event not found
+
+#### `GET /api/macro/impacts/{ticker}`
+Macro impacts and exposure profile for a specific company.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `since` | string | — | ISO 8601 | Impacts after this time |
+| `limit` | int | `50` | max `200` | Page size |
+| `offset` | int | `0` | — | Pagination offset |
+
+- **Response:** `{ exposure_profile, impacts[] }`
+
+### 1.18 Competitive Signal Layer
+
+#### `GET /api/admin/competitive/status`
+Return the current competitive signal layer enabled/disabled state.
+
+- **Response:** `{ competitive_enabled: bool, source: "default" | "risk_configs" }`
+
+#### `PUT /api/admin/competitive/toggle`
+Toggle the competitive signal layer on or off. Records an audit event.
+
+- **Body:** `{ enabled: bool, operator?: string (default "operator") }`
+- **Response:** `{ competitive_enabled, previous_enabled, toggled_by }`
+
+### 1.19 Patterns and Competitive Signals
+
+#### `GET /api/patterns/{ticker}`
+Historical patterns for a company.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `catalyst_type` | string | — | Filter by catalyst type |
+| `time_horizon` | string | — | Filter by time horizon |
+
+- **Response:** `{ ticker, patterns[], count }`
+
+#### `GET /api/patterns/{ticker}/competitors`
+Cross-company patterns showing how this company's catalysts affected competitors.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `catalyst_type` | string | — | Filter by catalyst type |
+| `time_horizon` | string | — | Filter by time horizon |
+
+- **Response:** `{ ticker, cross_company_patterns[], count }`
+
+#### `GET /api/patterns/{ticker}/competitive-signals`
+Recent competitive signals targeting this company (limit 100).
+
+- **Response:** `{ ticker, competitive_signals[], count }`
+
+#### `GET /api/patterns/{ticker}/decisions`
+Major corporate decision history with trend outcomes and pattern statistics.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `time_horizon` | string | — | Filter by time horizon |
+
+- **Response:** `{ ticker, decisions[], count }` — each decision includes `pattern_statistics[]`
+
+
+### 1.20 AI Agents
+
+#### `GET /api/agents`
+List all AI agent configurations.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `active_only` | bool | `false` | Only show active agents |
+
+#### `GET /api/agents/{agent_id}`
+Get a single agent configuration.
+
+- **Errors:** `404` — Agent not found
+
+#### `POST /api/agents` (201)
+Create a new user-defined agent.
+
+- **Body:** `AgentCreateBody`
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `name` | string | — | **Required** |
+| `slug` | string | auto-generated | URL-safe identifier |
+| `purpose` | string | `""` | Agent purpose |
+| `model_provider` | string | `"ollama"` | LLM provider |
+| `model_name` | string | `"llama3.1:8b"` | Model identifier |
+| `system_prompt` | string | `""` | System prompt |
+| `user_prompt_template` | string | `""` | User prompt template |
+| `prompt_version` | string | `""` | Prompt version tag |
+| `schema_version` | string | `"1.0.0"` | Output schema version |
+| `temperature` | float | `0.0` | Sampling temperature |
+| `max_tokens` | int | `32768` | Max output tokens |
+| `timeout_seconds` | int | `120` | Request timeout |
+| `max_retries` | int | `2` | Max retry attempts |
+
+#### `PUT /api/agents/{agent_id}`
+Update an agent configuration. Partial updates supported.
+
+- **Body:** `AgentUpdateBody` — all fields optional (same fields as create)
+- **Errors:** `400` — No fields to update; `404` — Agent not found
+
+#### `DELETE /api/agents/{agent_id}`
+Delete a user-created agent. System agents cannot be deleted.
+
+- **Errors:** `403` — Cannot delete system agents; `404` — Agent not found
+
+#### `GET /api/agents/{agent_id}/performance`
+Aggregated performance metrics for an agent.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | max `720` | Time window |
+
+- **Response:** `{ total_invocations, successes, failures, avg_duration_ms, p95_duration_ms, avg_confidence, avg_retries, total_input_tokens, total_output_tokens, success_rate }`
+
+#### `GET /api/agents/{agent_id}/performance/history`
+Hourly performance time-series for an agent.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | max `720` | Time window |
+
+- **Response:** Array of `{ hour, invocations, successes, avg_duration_ms, avg_confidence }`
+
+### 1.21 Agent Variants
+
+#### `GET /api/agents/{agent_id}/variants`
+List all variants for an agent, ordered by `created_at` ascending.
+
+#### `GET /api/agents/{agent_id}/variants/{variant_id}`
+Get a single variant.
+
+- **Errors:** `404` — Variant not found
+
+#### `POST /api/agents/{agent_id}/variants` (201)
+Create a new variant for an agent.
+
+- **Body:** `VariantCreateBody`
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `variant_name` | string | — | **Required** |
+| `variant_slug` | string | auto-generated | URL-safe identifier |
+| `description` | string | `""` | Variant description |
+| `model_provider` | string | `"ollama"` | LLM provider |
+| `model_name` | string | — | **Required.** Model identifier |
+| `system_prompt` | string | `""` | System prompt |
+| `user_prompt_template` | string | `""` | User prompt template |
+| `prompt_version` | string | `""` | Prompt version tag |
+| `temperature` | float | `0.0` | Sampling temperature |
+| `max_tokens` | int | `32768` | Max output tokens |
+| `context_window` | int | `0` | Context window size |
+| `input_token_limit` | int | `0` | Input token limit |
+| `token_budget` | int | `0` | Token budget |
+| `timeout_seconds` | int | `120` | Request timeout |
+| `max_retries` | int | `2` | Max retry attempts |
+
+- **Errors:** `409` — Duplicate variant slug
+
+#### `PUT /api/agents/{agent_id}/variants/{variant_id}`
+Partial update a variant.
+
+- **Body:** `VariantUpdateBody` — all fields optional
+- **Errors:** `400` — No fields to update; `404` — Variant not found
+
+#### `DELETE /api/agents/{agent_id}/variants/{variant_id}`
+Delete a variant. Cannot delete active variants.
+
+- **Errors:** `400` — Cannot delete active variant; `404` — Variant not found
+
+#### `POST /api/agents/{agent_id}/clone` (201)
+Clone an agent's configuration as a new variant with optional overrides.
+
+- **Body:** `VariantCloneBody { variant_name, variant_slug?, ...optional overrides }`
+- **Errors:** `404` — Agent not found; `409` — Duplicate slug
+
+#### `POST /api/agents/{agent_id}/variants/{variant_id}/clone` (201)
+Clone an existing variant as a new variant with optional overrides.
+
+- **Body:** `VariantCloneBody`
+- **Errors:** `404` — Source variant not found; `409` — Duplicate slug
+
+#### `POST /api/agents/{agent_id}/variants/{variant_id}/activate`
+Set a variant as the active variant for its agent. Deactivates any currently active variant in a transaction.
+
+- **Errors:** `404` — Variant not found
+
+#### `POST /api/agents/{agent_id}/variants/deactivate`
+Deactivate the currently active variant. Agent falls back to base configuration.
+
+#### `GET /api/agents/{agent_id}/variants/{variant_id}/performance`
+Aggregated performance metrics for a specific variant.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | max `720` | Time window |
+
+#### `GET /api/agents/{agent_id}/variants/{variant_id}/performance/history`
+Hourly performance time-series for a specific variant.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `hours` | int | `24` | max `720` | Time window |
+
+---
+
+## 2. Symbol Registry API
+
+Source: `services/symbol_registry/app.py` with routers from `exposure.py`, `competitors.py`, `competitor_inference.py`
+
+### 2.1 Health
+
+#### `GET /health`
+Liveness probe. Verifies database connectivity.
+
+- **Response:** `{"status": "ok"}`
+- **Errors:** `503` — Database unavailable
+
+### 2.2 Companies
+
+#### `POST /companies` (201)
+Create a new tracked company.
+
+- **Body:** `CompanyCreate`
+
+| Field | Type | Default | Constraints | Description |
+|-------|------|---------|-------------|-------------|
+| `ticker` | string | — | 1–10 uppercase letters | **Required.** Stock ticker |
+| `legal_name` | string | — | — | **Required.** Company name |
+| `exchange` | string | `null` | — | Stock exchange |
+| `sector` | string | `null` | — | Sector classification |
+| `industry` | string | `null` | — | Industry classification |
+| `market_cap_bucket` | string | `null` | — | Market cap bucket |
+
+- **Response:** `CompanyResponse { id, ticker, legal_name, exchange, sector, industry, market_cap_bucket, active }`
+- **Errors:** `409` — Company already exists; `422` — Invalid ticker format
+
+#### `GET /companies`
+List tracked companies.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `active` | bool | `true` | Filter by active status |
+
+- **Response:** Array of `CompanyResponse`
+
+#### `GET /companies/{company_id}`
+Get a single company.
+
+- **Errors:** `404` — Company not found
+
+#### `PUT /companies/{company_id}`
+Update a company.
+
+- **Body:** `CompanyCreate` (same as create)
+- **Errors:** `404` — Company not found
+
+### 2.3 Aliases
+
+#### `POST /companies/{company_id}/aliases` (201)
+Add an alias for a company.
+
+- **Body:** `{ alias: string, alias_type?: string (default "brand") }`
+- **Response:** `{ id, alias, alias_type }`
+
+#### `GET /companies/{company_id}/aliases`
+List aliases for a company.
+
+- **Response:** Array of `{ id, alias, alias_type }`
+
+### 2.4 Watchlists
+
+#### `POST /watchlists` (201)
+Create a new watchlist.
+
+- **Body:** `{ name: string, description?: string }`
+- **Errors:** `409` — Watchlist name already exists
+
+#### `GET /watchlists`
+List all watchlists.
+
+#### `POST /watchlists/{watchlist_id}/members/{company_id}` (201)
+Add a company to a watchlist.
+
+- **Errors:** `409` — Already a member; `404` — Watchlist or company not found
+
+#### `GET /watchlists/{watchlist_id}/members`
+List companies in a watchlist.
+
+- **Response:** Array of `CompanyResponse`
+
+### 2.5 Sources
+
+#### `POST /companies/{company_id}/sources` (201)
+Add a data source for a company.
+
+- **Body:** `SourceCreate`
+
+| Field | Type | Default | Constraints | Description |
+|-------|------|---------|-------------|-------------|
+| `source_type` | string | — | `market_api`, `news_api`, `filings_api`, `web_scrape`, `broker` | **Required** |
+| `source_name` | string | — | — | **Required** |
+| `config` | dict | `{}` | URLs validated for `base_url` | Source configuration |
+| `credibility_score` | float | `0.5` | — | Source credibility |
+| `retention_days` | int | `365` | — | Data retention period |
+| `access_policy` | string | `"internal"` | `internal`, `public`, `restricted` | Access policy |
+
+- **Errors:** `404` — Company not found; `422` — Invalid source_type or access_policy
+
+#### `GET /companies/{company_id}/sources`
+List sources for a company.
+
+### 2.6 Exposure Profiles
+
+#### `GET /companies/{company_id}/exposure`
+Get the current active exposure profile for a company.
+
+- **Response:** `ExposureProfileResponse { id, company_id, geographic_revenue_mix, supply_chain_regions, key_input_commodities, regulatory_jurisdictions, market_position_tier, export_dependency_pct, source, confidence, version, active, created_at, updated_at }`
+- **Errors:** `404` — No active exposure profile found
+
+#### `PUT /companies/{company_id}/exposure`
+Create or update an exposure profile. Archives the previous active version.
+
+- **Body:** `ExposureProfileCreate`
+
+| Field | Type | Default | Constraints | Description |
+|-------|------|---------|-------------|-------------|
+| `geographic_revenue_mix` | dict[str, float] | `{}` | — | Region to revenue percentage |
+| `supply_chain_regions` | list[str] | `[]` | — | Supply chain regions |
+| `key_input_commodities` | list[str] | `[]` | — | Key input commodities |
+| `regulatory_jurisdictions` | list[str] | `[]` | — | Regulatory jurisdictions |
+| `market_position_tier` | string | `"regional"` | `global_leader`, `multinational`, `regional`, `domestic` | Market position |
+| `export_dependency_pct` | float | `0.0` | 0.0–1.0 | Export dependency |
+| `source` | string | `"manual"` | `manual`, `inferred` | Data source |
+| `confidence` | float | `1.0` | 0.0–1.0 | Confidence score |
+
+- **Errors:** `404` — Company not found
+
+#### `GET /companies/{company_id}/exposure/history`
+Get all exposure profile versions for a company, ordered by version descending.
+
+### 2.7 Competitor Relationships
+
+#### `POST /companies/{company_id}/competitors` (201)
+Create a competitor relationship. Records an audit event.
+
+- **Body:** `CompetitorRelationshipCreate`
+
+| Field | Type | Default | Constraints | Description |
+|-------|------|---------|-------------|-------------|
+| `company_b_id` | string | — | UUID | **Required.** Other company |
+| `relationship_type` | string | — | `direct_rival`, `same_sector`, `overlapping_products`, `supply_chain_adjacent` | **Required** |
+| `strength` | float | `0.5` | 0–1 | Relationship strength |
+| `bidirectional` | bool | `true` | — | Bidirectional relationship |
+| `source` | string | `"manual"` | `manual`, `inferred` | Data source |
+
+- **Errors:** `400` — Self-reference; `404` — Company not found; `409` — Relationship already exists
+
+#### `GET /companies/{company_id}/competitors`
+List active competitor relationships, enriched with ticker and legal_name of the other company.
+
+- **Errors:** `404` — Company not found
+
+#### `PUT /companies/{company_id}/competitors/{relationship_id}`
+Update a competitor relationship. Records an audit event with previous state.
+
+- **Body:** `CompetitorRelationshipCreate`
+- **Errors:** `404` — Relationship not found
+
+#### `DELETE /companies/{company_id}/competitors/{relationship_id}`
+Soft-delete a competitor relationship (sets `active=false`). Records an audit event.
+
+- **Errors:** `404` — Active relationship not found
+
+### 2.8 Competitor Inference
+
+#### `POST /companies/{company_id}/competitors/infer`
+Auto-infer competitor relationships based on sector/industry match and document co-mention frequency.
+
+Strength formula: `0.3 * sector_match + 0.7 * normalized_co_mention_count`
+
+Upserts relationships with `source='inferred'` and `relationship_type='same_sector'`.
+
+- **Response:** Array of `CompetitorRelationship` objects sorted by strength descending
+- **Errors:** `400` — Company missing sector or industry; `404` — Company not found
+
+---
+
+## 3. Trading Engine API
+
+Source: `services/trading/app.py`
+
+### 3.1 Health and Readiness
+
+#### `GET /health`
+Liveness probe.
+
+- **Response:** `{"status": "ok"}`
+
+#### `GET /ready`
+Readiness probe — reports whether the engine is running.
+
+- **Response:** `{"ready": bool}`
+
+### 3.2 Debug
+
+#### `GET /api/trading/debug`
+Diagnostic endpoint showing engine internals for troubleshooting.
+
+- **Response:** `{ running, has_pool, has_redis, config_enabled, polling_interval, last_poll, portfolio_state: { active_pool, reserve_pool, total_value, open_positions }, risk_tier, tasks, processed_rec_count }`
+
+### 3.3 Engine Status and Control
+
+#### `GET /api/trading/status`
+Return current engine state.
+
+- **Response:** `{ enabled, paused, risk_tier, circuit_breaker_status, active_pool, reserve_pool, portfolio_heat, open_positions, last_decision_at }`
+- **Errors:** `503` — Engine not initialised
+
+#### `PUT /api/trading/config`
+Update trading engine configuration. Returns previous and new values for audit trail.
+
+- **Body:** `ConfigUpdateRequest`
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `enabled` | bool | Enable/disable engine |
+| `risk_tier` | string | Risk tier level |
+| `reserve_siphon_pct` | float | Reserve pool siphon percentage |
+| `polling_interval_seconds` | int | Polling interval |
+| `absolute_position_cap` | float | Max position value |
+| `active_pool_minimum` | float | Minimum active pool |
+| `micro_trading_enabled` | bool | Enable micro trades |
+| `max_open_positions` | int | Max concurrent positions |
+
+All fields optional. Only provided fields are updated.
+
+- **Response:** `{ previous, updated, change_source: "api", changed_at }`
+- **Errors:** `503` — Engine not initialised
+
+#### `POST /api/trading/pause`
+Pause the trading engine.
+
+- **Response:** `{"paused": true}`
+
+#### `POST /api/trading/resume`
+Resume the trading engine.
+
+- **Response:** `{"paused": false}`
+
+#### `POST /api/trading/reset`
+Full paper trading reset: liquidate broker positions, cancel orders, clear trading state, reset capital.
+
+- **Body:** `{ initial_capital?: float (default 0.0) }` — if 0, uses broker balance or defaults to 100,000
+- **Response:** `{ reset: true, initial_capital, active_pool, reserve_pool, broker: { orders_cancelled, positions_closed, portfolio_value, cash, buying_power } }`
+- **Errors:** `503` — Engine not initialised; `500` — Database reset failed
+
+### 3.4 Decision Audit Trail
+
+#### `GET /api/trading/decisions`
+Return recent trading decisions from the database.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `ticker` | string | — | — | Filter by ticker |
+| `decision` | string | — | — | Filter by decision type |
+| `is_micro_trade` | bool | — | — | Filter micro trades |
+| `limit` | int | `50` | max `200` | Page size |
+| `offset` | int | `0` | — | Pagination offset |
+
+### 3.5 Performance Metrics
+
+#### `GET /api/trading/metrics`
+Return current performance metrics.
+
+- **Response:** `{ total_portfolio_value, active_pool, reserve_pool, unrealized_pnl, realized_pnl, daily_pnl, win_rate, profit_factor, sharpe_ratio, max_drawdown, portfolio_heat }`
+- **Errors:** `503` — Engine not initialised
+
+#### `GET /api/trading/metrics/history`
+Return historical daily portfolio snapshots.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `limit` | int | `30` | max `365` | Max snapshots |
+
+### 3.6 Backtesting
+
+#### `POST /api/trading/backtest`
+Launch a backtest run asynchronously.
+
+- **Body:** `BacktestRequest`
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `start_date` | string | — | **Required.** ISO date (YYYY-MM-DD) |
+| `end_date` | string | — | **Required.** ISO date (YYYY-MM-DD) |
+| `initial_capital` | float | `500.0` | Starting capital |
+| `risk_tier` | string | `"moderate"` | Risk tier for backtest |
+
+- **Response:** `{ id: string, status: "running" }`
+- **Errors:** `503` — Engine not initialised
+
+#### `GET /api/trading/backtest/{backtest_id}`
+Retrieve backtest results.
+
+- **Response:** `{ id, start_date, end_date, initial_capital, risk_tier, config, total_return, sharpe_ratio, max_drawdown, win_rate, profit_factor, trade_count, equity_curve[], trades[], status, completed_at, created_at }`
+- Status values: `running`, `completed`, `not_found`, `pending`
+
+### 3.7 Notifications
+
+#### `GET /api/trading/notifications/config`
+Return current notification configuration.
+
+- **Response:** `{ sms_enabled, email_enabled, phone_number, email_recipient }`
+- **Errors:** `503` — Engine not initialised
+
+#### `PUT /api/trading/notifications/config`
+Update notification preferences.
+
+- **Body:** `NotificationConfigRequest`
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `sms_enabled` | bool | Enable SMS notifications |
+| `email_enabled` | bool | Enable email notifications |
+| `phone_number` | string | SMS phone number |
+| `email_recipient` | string | Email recipient address |
+
+All fields optional.
+
+- **Errors:** `503` — Engine not initialised
+
+#### `GET /api/trading/notifications/history`
+Return recent notifications.
+
+| Parameter | Type | Default | Constraints | Description |
+|-----------|------|---------|-------------|-------------|
+| `limit` | int | `50` | max `200` | Max results |
+
+### 3.8 Override Orders
+
+#### `POST /api/trading/override/order` (202)
+Submit a manual override order to the broker queue. Auto-registers untracked tickers in the Symbol Registry.
+
+- **Body:** `OverrideOrderRequest`
+
+| Field | Type | Default | Constraints | Description |
+|-------|------|---------|-------------|-------------|
+| `ticker` | string | — | 1–10 alphabetic chars | **Required.** Stock ticker |
+| `side` | string | — | `buy` or `sell` | **Required.** Order side |
+| `quantity` | float | — | Must be positive | **Required.** Share quantity |
+| `order_type` | string | `"market"` | `market`, `limit`, `stop`, `stop_limit` | Order type |
+| `limit_price` | float | `null` | Required for `limit`/`stop_limit` | Limit price |
+| `stop_price` | float | `null` | Required for `stop`/`stop_limit` | Stop price |
+
+- **Response:** `OverrideOrderResponse { job_id, status: "queued", ticker, side, quantity, auto_registered }`
+- **Errors:** `503` — Engine not initialised or broker queue unavailable; `422` — Validation errors
+
+---
+
+## 4. Risk Engine API
+
+Source: `services/risk/app.py`
+
+### 4.1 Health
+
+#### `GET /health`
+Liveness probe.
+
+- **Response:** `{"status": "ok"}`
+
+### 4.2 Order Evaluation
+
+#### `POST /evaluate`
+Evaluate a proposed order against risk rules.
+
+- **Body:** `EvaluateRequest`
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `order` | ProposedOrder | **Required.** The order to evaluate |
+| `config` | PortfolioRiskConfig | Optional. Risk config (uses defaults if null) |
+| `state` | AccountRiskState | Optional. Current account state |
+
+**ProposedOrder schema:**
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `recommendation_id` | string | `null` | Source recommendation |
+| `ticker` | string | — | **Required.** Stock ticker |
+| `sector` | string | `""` | Company sector |
+| `action` | string | `"buy"` | `buy` or `sell` |
+| `quantity` | float | `0.0` | Share quantity |
+| `estimated_value` | float | `0.0` | Estimated order value |
+| `confidence` | float | `0.0` | Recommendation confidence |
+
+- **Response:** `RiskEvaluation { evaluation_id, recommendation_id, ticker, eligible, allowed_mode, checks[], rejection_reasons[], config_snapshot, state_snapshot, evaluated_at }`
+
+### 4.3 Approvals
+
+#### `GET /approvals/pending`
+List pending approval requests.
+
+- **Response:** Array of approval request objects (serialized via `to_dict()`)
+- **Errors:** `503` — Database not ready
+
+#### `GET /approvals/{approval_id}`
+Get a single approval request.
+
+- **Errors:** `404` — Approval not found; `503` — Database not ready
+
+#### `POST /approvals/{approval_id}/review`
+Approve or reject a pending approval request.
+
+- **Body:** `ReviewRequest`
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `approved` | bool | — | **Required.** Approve or reject |
+| `reviewed_by` | string | `"operator"` | Reviewer identity |
+| `review_note` | string | `""` | Optional review note |
+
+- **Response:** `{ approval_id, status }`
+- **Errors:** `404` — Approval not found or no longer pending; `503` — Database not ready
+
+### 4.4 Approval Expiration
+
+#### `POST /approvals/expire`
+Expire stale approvals that have passed their expiration time.
+
+- **Response:** `{ expired: int, items: [] }`
+- **Errors:** `503` — Database not ready
\ No newline at end of file
diff --git a/docs/architecture-data-pipeline.md b/docs/architecture-data-pipeline.md
new file mode 100644
index 0000000..22b40ed
--- /dev/null
+++ b/docs/architecture-data-pipeline.md
@@ -0,0 +1,274 @@
+# Data Pipeline Architecture — Stonks Oracle
+
+This document describes the end-to-end data pipeline from external data sources through signal processing to trade execution. The pipeline is queue-driven, with Redis lists connecting each stage and PostgreSQL/MinIO providing durable storage at every step.
+
+All queue names follow the convention `stonks:queue:` (see `services/shared/redis_keys.py`). Dead-letter queues mirror the pattern as `stonks:dlq:`.
+
+## Pipeline Overview
+
+```mermaid
+flowchart TB
+ %% ── External Data Sources ─────────────────────────────────────
+ subgraph sources ["External Data Sources"]
+ direction LR
+ polygon["Polygon.io
News, Market Bars,
Grouped Daily"]
+ sec["SEC EDGAR
10-K, 10-Q Filings"]
+ macro_src["Macro News APIs
Geopolitical &
Economic Events"]
+ market_src["Market Data API
Intraday Bars,
Grouped Daily"]
+ end
+
+ %% ── Scheduler ─────────────────────────────────────────────────
+ scheduler["Scheduler
services.scheduler.app
Cadence polling, rate limiting,
backoff & stale recovery"]
+
+ sources -.->|"API polling
on cadence"| scheduler
+
+ %% ── Ingestion Queue ───────────────────────────────────────────
+ q_ingestion[["stonks:queue:ingestion"]]
+ scheduler -->|"rpush job"| q_ingestion
+
+ %% ── Ingestion Worker ──────────────────────────────────────────
+ ingestion["Ingestion
services.ingestion.worker
Adapter dispatch, dedupe,
raw artifact upload"]
+
+ q_ingestion -->|"lpop"| ingestion
+
+ %% ── Raw Storage ───────────────────────────────────────────────
+ minio_raw[("MinIO
Raw Artifacts
JSON / HTML")]
+ pg_docs[("PostgreSQL
documents,
ingestion_runs")]
+ redis_dedupe[("Redis
Dedupe Markers
stonks:dedupe:*")]
+
+ ingestion -->|"upload raw payload"| minio_raw
+ ingestion -->|"persist metadata"| pg_docs
+ ingestion -->|"set content hash"| redis_dedupe
+
+ %% ── Parsing Queue ─────────────────────────────────────────────
+ q_parsing[["stonks:queue:parsing"]]
+ ingestion -->|"rpush
(news, filings,
web_scrape)"| q_parsing
+
+ %% ── Parser Worker ─────────────────────────────────────────────
+ parser["Parser
services.parser.worker
HTML parsing, quality scoring,
company mention detection"]
+
+ q_parsing -->|"lpop"| parser
+
+ minio_norm[("MinIO
Normalized Text
Parser Output JSON")]
+ parser -->|"upload normalized text"| minio_norm
+ parser -->|"update document status,
insert mentions"| pg_docs
+```
+
+## Three Signal Layers
+
+The parser routes documents into two extraction paths based on `document_type`. All three signal layers converge at the aggregation stage through the shared `WeightedSignal` abstraction.
+
+```mermaid
+flowchart TB
+ %% ── Parser Output ─────────────────────────────────────────────
+ parser(("Parser"))
+
+ %% ── Extraction Queues ─────────────────────────────────────────
+ q_extraction[["stonks:queue:extraction"]]
+ q_macro[["stonks:queue:macro_classification"]]
+
+ parser -->|"rpush
(standard docs)"| q_extraction
+ parser -->|"rpush
(macro_event docs)"| q_macro
+
+ %% ── Extractor Worker ──────────────────────────────────────────
+ subgraph extractor_svc ["Extractor Service"]
+ direction TB
+ ext_main["Extractor
services.extractor.main
Alternates between queues
(2 extraction : 1 macro)"]
+ end
+
+ q_extraction -->|"lpop"| ext_main
+ q_macro -->|"lpop"| ext_main
+
+ %% ── Ollama LLM ───────────────────────────────────────────────
+ ollama["Ollama
LLM Inference
document-extractor agent
event-classifier agent"]
+ ext_main <-->|"HTTP /api/generate"| ollama
+
+ %% ── Signal Layer 1: Company ───────────────────────────────────
+ subgraph layer1 ["Layer 1 — Company Signals"]
+ direction LR
+ di["document_intelligence
document_impact_records"]
+ end
+
+ ext_main -->|"persist extraction
(standard docs)"| di
+
+ %% ── Signal Layer 2: Macro ─────────────────────────────────────
+ subgraph layer2 ["Layer 2 — Macro Signals"]
+ direction LR
+ ge["global_events"]
+ mir["macro_impact_records
per-company interpolation"]
+ ge --> mir
+ end
+
+ ext_main -->|"classify & persist
(macro_event docs)"| ge
+ ext_main -->|"compute_macro_impact
for all tracked companies"| mir
+
+ %% ── Aggregation Queue ─────────────────────────────────────────
+ q_agg[["stonks:queue:aggregation"]]
+ ext_main -->|"rpush
(per ticker)"| q_agg
+
+ %% ── Aggregation Worker ────────────────────────────────────────
+ aggregation["Aggregation
services.aggregation.main
Trend windows, scoring,
contradiction detection"]
+
+ q_agg -->|"lpop"| aggregation
+
+ %% ── Signal Layer 3: Competitive ──────────────────────────────
+ subgraph layer3 ["Layer 3 — Competitive Signals"]
+ direction LR
+ pm["pattern_matcher
historical patterns"]
+ sp["signal_propagation
cross-company signals"]
+ csr["competitive_signal_records"]
+ pm --> sp --> csr
+ end
+
+ aggregation -->|"trigger_signal_propagation
(when competitive_enabled)"| layer3
+
+ %% ── All layers merge ──────────────────────────────────────────
+ pg_trends[("PostgreSQL
trend_windows,
trend_history,
trend_projections")]
+
+ di -->|"WeightedSignal"| aggregation
+ mir -->|"WeightedSignal"| aggregation
+ csr -->|"WeightedSignal"| aggregation
+ aggregation -->|"persist trend summaries"| pg_trends
+```
+
+## Recommendation → Trading → Broker
+
+```mermaid
+flowchart TB
+ %% ── Recommendation Queue ──────────────────────────────────────
+ q_rec[["stonks:queue:recommendation"]]
+ aggregation(("Aggregation")) -->|"rpush
(ticker + window,
dedup 5 min TTL)"| q_rec
+
+ %% ── Recommendation Worker ─────────────────────────────────────
+ recommendation["Recommendation
services.recommendation.main
Eligibility, suppression,
thesis generation"]
+
+ q_rec -->|"lpop"| recommendation
+
+ ollama_thesis["Ollama
thesis-rewriter agent
(optional LLM rewrite)"]
+ recommendation <-->|"rewrite thesis
(trading-eligible only)"| ollama_thesis
+
+ pg_recs[("PostgreSQL
recommendations,
recommendation_evidence,
risk_evaluations")]
+ recommendation -->|"persist recommendation
+ evidence + risk eval"| pg_recs
+
+ %% ── Trading Engine ────────────────────────────────────────────
+ subgraph trading_loop ["Trading Engine Decision Loop"]
+ direction TB
+ poll["Poll recommendations
action IN (buy, sell)
mode IN (paper, live)
generated_at > last_poll"]
+ dedup_check["Redis dedup check
stonks:dedupe:trading:*"]
+ evaluate["evaluate_recommendation
Circuit breaker check
Trading window check
Confidence gate
Sector exposure check
Correlation check
Earnings blackout"]
+ size["Position sizing
Kelly criterion,
risk tier limits"]
+ decide{{"Decision"}}
+ poll --> dedup_check --> evaluate --> size --> decide
+ end
+
+ pg_recs -->|"SELECT recent
recommendations"| poll
+
+ %% ── Broker Queue ──────────────────────────────────────────────
+ q_broker[["stonks:queue:broker_orders"]]
+ decide -->|"act → rpush
order job"| q_broker
+ decide -->|"skip → persist
decision only"| pg_decisions
+
+ pg_decisions[("PostgreSQL
trading_decisions")]
+
+ %% ── Broker Adapter ────────────────────────────────────────────
+ broker["Broker Adapter
services.adapters.broker_service
Risk evaluation, idempotency,
order submission, fill tracking"]
+
+ q_broker -->|"lpop"| broker
+
+ %% ── Risk Engine ───────────────────────────────────────────────
+ risk["Risk Engine
services.risk.app
POST /evaluate
Approval workflow"]
+ broker <-->|"evaluate order"| risk
+
+ %% ── Alpaca ────────────────────────────────────────────────────
+ alpaca["Alpaca
Paper Trading API
Order submission,
position sync"]
+ broker <-->|"submit order /
sync positions"| alpaca
+
+ pg_orders[("PostgreSQL
orders, order_events,
positions,
portfolio_snapshots")]
+ broker -->|"persist order,
events, positions"| pg_orders
+
+ %% ── Notifications ─────────────────────────────────────────────
+ subgraph notifications ["Notifications"]
+ direction LR
+ sns["AWS SNS
SMS alerts"]
+ gmail["Gmail SMTP
Email alerts"]
+ end
+
+ trading_loop -->|"circuit breaker trips,
order fills,
stop-loss triggers"| notifications
+```
+
+## Analytical Branch — Lake Publisher
+
+The lake publisher runs as a separate worker, consuming from its own queue and writing partitioned Parquet fact tables to MinIO for analytical queries.
+
+```mermaid
+flowchart LR
+ %% ── Lake Publish Queue ────────────────────────────────────────
+ q_lake[["stonks:queue:lake_publish"]]
+
+ various(("Various Services
ingestion, extractor,
recommendation,
broker adapter"))
+ various -->|"enqueue_lake_job"| q_lake
+
+ %% ── Lake Publisher Worker ─────────────────────────────────────
+ lake["Lake Publisher
services.lake_publisher.jobs
Transforms operational data
into analytical facts"]
+
+ q_lake -->|"lpop"| lake
+
+ pg_source[("PostgreSQL
Operational Tables
documents, extractions,
orders, positions, events")]
+ lake -->|"query source data"| pg_source
+
+ %% ── MinIO Parquet ─────────────────────────────────────────────
+ minio_lake[("MinIO
Lakehouse Bucket
Partitioned Parquet
/year=/month=/day=")]
+ lake -->|"write Parquet files"| minio_lake
+
+ %% ── Trino ─────────────────────────────────────────────────────
+ trino["Trino
SQL Query Engine
Hive connector → MinIO"]
+ minio_lake -->|"read via
Hive Metastore"| trino
+
+ hive["Hive Metastore
Schema catalog"]
+ trino <-->|"table metadata"| hive
+ hive -->|"location refs"| minio_lake
+
+ %% ── Visualization ─────────────────────────────────────────────
+ superset["Superset
Dashboards &
SQL Lab"]
+ dashboard["React Dashboard
frontend
Charts, portfolio,
recommendations"]
+ query_api["Query API
services.api.app"]
+
+ trino --> superset
+ trino --> query_api
+ query_api --> dashboard
+```
+
+## Complete Queue Topology
+
+| Queue | Full Key | Producer(s) | Consumer |
+|-------|----------|-------------|----------|
+| Ingestion | `stonks:queue:ingestion` | Scheduler | Ingestion Worker |
+| Parsing | `stonks:queue:parsing` | Ingestion Worker | Parser Worker |
+| Extraction | `stonks:queue:extraction` | Parser (standard docs) | Extractor Worker |
+| Macro Classification | `stonks:queue:macro_classification` | Parser (macro_event docs), Scheduler | Extractor Worker |
+| Aggregation | `stonks:queue:aggregation` | Extractor Worker | Aggregation Worker |
+| Recommendation | `stonks:queue:recommendation` | Aggregation Worker | Recommendation Worker |
+| Broker Orders | `stonks:queue:broker_orders` | Trading Engine, Trading API (manual overrides) | Broker Adapter |
+| Lake Publish | `stonks:queue:lake_publish` | Various services | Lake Publisher |
+
+Dead-letter queues follow the pattern `stonks:dlq:` and are populated when a job exhausts its retry budget.
+
+## Data Store Summary
+
+| Store | Role | Key Tables / Buckets |
+|-------|------|---------------------|
+| **PostgreSQL** | Structured operational data | `documents`, `document_intelligence`, `document_impact_records`, `global_events`, `macro_impact_records`, `competitive_signal_records`, `trend_windows`, `trend_history`, `trend_projections`, `recommendations`, `recommendation_evidence`, `risk_evaluations`, `orders`, `order_events`, `positions`, `portfolio_snapshots`, `trading_decisions` |
+| **Redis** | Queues, dedup markers, rate limits, circuit breaker state | `stonks:queue:*`, `stonks:dedupe:*`, `stonks:ratelimit:*`, `stonks:trading:circuit_breaker:*`, `stonks:dlq:*` |
+| **MinIO** | Object storage for raw artifacts, normalized text, and analytical Parquet files | Raw artifacts bucket, normalized text bucket, lakehouse bucket (partitioned Parquet) |
+
+## External Integration Points
+
+| Integration | Service | Protocol | Purpose |
+|-------------|---------|----------|---------|
+| **Polygon.io** | Ingestion (via adapters) | HTTPS REST | News articles, market bars, grouped daily data |
+| **SEC EDGAR** | Ingestion (via FilingsDataAdapter) | HTTPS REST | 10-K, 10-Q filings |
+| **Ollama** | Extractor, Recommendation | HTTP `/api/generate` | LLM inference for document extraction, event classification, thesis rewriting |
+| **Alpaca** | Broker Adapter | HTTPS REST | Paper trading order submission, position sync, account state |
+| **AWS SNS** | Trading Engine (notifications) | boto3 SDK | SMS alerts for circuit breaker trips, order fills, stop-loss triggers |
+| **Gmail** | Trading Engine (notifications) | SMTP (port 587 STARTTLS) | Email alerts for trading events |
+| **Trino** | Query API, Superset | JDBC / HTTP | SQL queries over lakehouse Parquet files |
diff --git a/docs/architecture-docker-compose.md b/docs/architecture-docker-compose.md
new file mode 100644
index 0000000..88d811c
--- /dev/null
+++ b/docs/architecture-docker-compose.md
@@ -0,0 +1,322 @@
+# Docker Compose Architecture — Stonks Oracle
+
+This document describes the Docker Compose deployment topology for Stonks Oracle, derived from the `docker-compose.yml` file at the repository root.
+
+All containers run on a single Docker network created by Compose. Infrastructure services (PostgreSQL, Redis, MinIO, Ollama, Trino, Hive Metastore, Superset) start first, and application services wait for their dependencies via `depends_on` with health check conditions.
+
+## Container Topology Diagram
+
+```mermaid
+graph TB
+ %% ── Host machine ──────────────────────────────────────────────
+ host((Host Machine))
+
+ %% ── .env file ─────────────────────────────────────────────────
+ envfile[".env file
MARKET_DATA_API_KEY
BROKER_API_KEY
BROKER_API_SECRET
BROKER_BASE_URL"]
+
+ %% ── Docker Compose default network ────────────────────────────
+ subgraph network ["Docker Compose Network (default)"]
+ direction TB
+
+ %% ── Infrastructure Containers ─────────────────────────────
+ subgraph infra ["Infrastructure Containers"]
+ direction LR
+ postgres[("postgres
postgres:16-alpine
host :5432 → :5432")]
+ redis[("redis
redis:7-alpine
host :6379 → :6379")]
+ minio[("minio
minio/minio:latest
host :9000 → :9000
host :9001 → :9001")]
+ ollama[("ollama
ollama/ollama:latest
host :11434 → :11434")]
+ end
+
+ subgraph infra_init ["Infrastructure Init"]
+ minio_init["minio-init
minio/mc:latest
Creates buckets on startup"]
+ end
+
+ subgraph analytics ["Analytics Containers"]
+ direction LR
+ hive_metastore["hive-metastore
apache/hive:4.0.0
host :9083 → :9083"]
+ trino["trino
trinodb/trino:latest
host :8080 → :8080"]
+ superset["superset
apache/superset:latest
host :8088 → :8088"]
+ end
+
+ %% ── Application Containers ────────────────────────────────
+
+ subgraph api_tier ["API Tier"]
+ direction LR
+ query_api["query-api
docker/Dockerfile
uvicorn services.api.app
host :8004 → :8000"]
+ symbol_registry["symbol-registry
docker/Dockerfile
uvicorn services.symbol_registry.app
host :8001 → :8000"]
+ end
+
+ subgraph frontend_tier ["Frontend Tier"]
+ dashboard["dashboard
frontend/Dockerfile
nginx on :8080
host :3000 → :8080"]
+ end
+
+ subgraph trading_tier ["Trading Tier"]
+ direction LR
+ trading_engine["trading-engine
docker/Dockerfile
uvicorn services.trading.app
host :8002 → :8000"]
+ risk_engine["risk-engine
docker/Dockerfile
uvicorn services.risk.app
host :8003 → :8000"]
+ broker_adapter["broker-adapter
docker/Dockerfile
python -m services.adapters.broker_service
no host port"]
+ end
+
+ subgraph orchestration_tier ["Orchestration Tier"]
+ scheduler["scheduler
docker/Dockerfile.scheduler
no host port"]
+ end
+
+ subgraph processing_tier ["Processing Tier (pipeline workers)"]
+ direction LR
+ ingestion["ingestion
docker/Dockerfile
python -m services.ingestion.worker
no host port"]
+ parser["parser
docker/Dockerfile
python -m services.parser.worker
no host port"]
+ extractor["extractor
docker/Dockerfile
python -m services.extractor.main
no host port"]
+ aggregation["aggregation
docker/Dockerfile
python -m services.aggregation.main
no host port"]
+ recommendation["recommendation
docker/Dockerfile
python -m services.recommendation.main
no host port"]
+ end
+
+ subgraph analytics_worker ["Analytics Worker"]
+ lake_publisher["lake-publisher
docker/Dockerfile
python -m services.lake_publisher.jobs
no host port"]
+ end
+ end
+
+ %% ── Host port access ──────────────────────────────────────────
+ host -->|":5432"| postgres
+ host -->|":6379"| redis
+ host -->|":9000 / :9001"| minio
+ host -->|":11434"| ollama
+ host -->|":8080"| trino
+ host -->|":9083"| hive_metastore
+ host -->|":8088"| superset
+ host -->|":8001"| symbol_registry
+ host -->|":8004"| query_api
+ host -->|":8002"| trading_engine
+ host -->|":8003"| risk_engine
+ host -->|":3000"| dashboard
+
+ %% ── .env injection ────────────────────────────────────────────
+ envfile -.->|"env_file: .env"| ingestion
+ envfile -.->|"env_file: .env"| broker_adapter
+ envfile -.->|"env_file: .env"| trading_engine
+
+ %% ── Styles ────────────────────────────────────────────────────
+ classDef infraSvc fill:#95a5a6,stroke:#717d7e,color:#fff
+ classDef analyticsSvc fill:#e74c3c,stroke:#a93226,color:#fff
+ classDef apiSvc fill:#4a90d9,stroke:#2c5f8a,color:#fff
+ classDef frontendSvc fill:#50c878,stroke:#2e7d46,color:#fff
+ classDef tradingSvc fill:#e8a838,stroke:#b07d1a,color:#fff
+ classDef orchSvc fill:#1abc9c,stroke:#148f77,color:#fff
+ classDef processSvc fill:#9b59b6,stroke:#6c3483,color:#fff
+ classDef initSvc fill:#bdc3c7,stroke:#7f8c8d,color:#333
+ classDef envSvc fill:#f5f5dc,stroke:#999,color:#333
+
+ class postgres,redis,minio,ollama infraSvc
+ class hive_metastore,trino,superset,lake_publisher analyticsSvc
+ class query_api,symbol_registry apiSvc
+ class dashboard frontendSvc
+ class trading_engine,risk_engine,broker_adapter tradingSvc
+ class scheduler orchSvc
+ class ingestion,parser,extractor,aggregation,recommendation processSvc
+ class minio_init initSvc
+ class envfile envSvc
+```
+
+## Dependency Graph
+
+The following diagram shows `depends_on` relationships and health check conditions. Solid arrows indicate `condition: service_healthy` (the dependent waits for the health check to pass). Dashed arrows indicate `condition: service_started` (the dependent waits only for the container to start).
+
+```mermaid
+graph LR
+ %% ── Infrastructure health checks ──────────────────────────────
+ postgres[("postgres
pg_isready -U stonks")]
+ redis[("redis
redis-cli ping")]
+ minio[("minio
mc ready local")]
+ ollama[("ollama
no health check")]
+
+ %% ── Analytics dependencies ────────────────────────────────────
+ hive["hive-metastore"] -->|started| minio
+ trino["trino"] -->|started| minio
+ trino -->|started| hive
+ superset["superset"] -->|started| trino
+ minio_init["minio-init"] -->|healthy| minio
+
+ %% ── Application depends_on (healthy) ──────────────────────────
+ scheduler["scheduler"] -->|healthy| postgres
+ scheduler -->|healthy| redis
+
+ symbol_registry["symbol-registry"] -->|healthy| postgres
+
+ ingestion["ingestion"] -->|healthy| postgres
+ ingestion -->|healthy| redis
+ ingestion -->|healthy| minio
+
+ parser["parser"] -->|healthy| postgres
+ parser -->|healthy| redis
+
+ extractor["extractor"] -->|healthy| postgres
+ extractor -->|healthy| redis
+ extractor -.->|started| ollama
+
+ aggregation["aggregation"] -->|healthy| postgres
+ aggregation -->|healthy| redis
+
+ recommendation["recommendation"] -->|healthy| postgres
+ recommendation -->|healthy| redis
+
+ trading_engine["trading-engine"] -->|healthy| postgres
+ trading_engine -->|healthy| redis
+
+ risk_engine["risk-engine"] -->|healthy| postgres
+
+ broker_adapter["broker-adapter"] -->|healthy| postgres
+ broker_adapter -->|healthy| redis
+
+ lake_publisher["lake-publisher"] -->|healthy| postgres
+ lake_publisher -->|healthy| minio
+
+ query_api["query-api"] -->|healthy| postgres
+ query_api -->|healthy| redis
+ query_api -->|healthy| minio
+
+ dashboard["dashboard"] -->|healthy| query_api
+
+ %% ── Styles ────────────────────────────────────────────────────
+ classDef infraSvc fill:#95a5a6,stroke:#717d7e,color:#fff
+ classDef appSvc fill:#4a90d9,stroke:#2c5f8a,color:#fff
+ classDef analyticsSvc fill:#e74c3c,stroke:#a93226,color:#fff
+ classDef initSvc fill:#bdc3c7,stroke:#7f8c8d,color:#333
+
+ class postgres,redis,minio,ollama infraSvc
+ class scheduler,symbol_registry,ingestion,parser,extractor,aggregation,recommendation,trading_engine,risk_engine,broker_adapter,lake_publisher,query_api,dashboard appSvc
+ class hive,trino,superset analyticsSvc
+ class minio_init initSvc
+```
+
+## Named Volumes
+
+Docker Compose defines five named volumes for persistent data:
+
+```mermaid
+graph LR
+ pgdata["📦 pgdata"]
+ miniodata["📦 miniodata"]
+ ollama_models["📦 ollama_models"]
+ hive_data["📦 hive_data"]
+ superset_data["📦 superset_data"]
+
+ pgdata -->|"/var/lib/postgresql/data"| postgres[("postgres")]
+ miniodata -->|"/data"| minio[("minio")]
+ ollama_models -->|"/root/.ollama"| ollama[("ollama")]
+ hive_data -->|"/opt/hive/data"| hive["hive-metastore"]
+ superset_data -->|"/app/superset_home"| superset["superset"]
+
+ classDef volStyle fill:#f5f5dc,stroke:#999,color:#333
+ classDef svcStyle fill:#95a5a6,stroke:#717d7e,color:#fff
+
+ class pgdata,miniodata,ollama_models,hive_data,superset_data volStyle
+ class postgres,minio,ollama,hive,superset svcStyle
+```
+
+| Volume | Mount Point | Container | Purpose |
+|--------|-------------|-----------|---------|
+| `pgdata` | `/var/lib/postgresql/data` | postgres | PostgreSQL database files |
+| `miniodata` | `/data` | minio | MinIO object storage data |
+| `ollama_models` | `/root/.ollama` | ollama | Downloaded LLM model weights |
+| `hive_data` | `/opt/hive/data` | hive-metastore | Hive Metastore embedded Derby DB |
+| `superset_data` | `/app/superset_home` | superset | Superset configuration and metadata |
+
+### Bind Mounts
+
+In addition to named volumes, several containers use bind mounts for configuration files:
+
+| Host Path | Mount Point | Container | Mode |
+|-----------|-------------|-----------|------|
+| `./infra/migrations/` | `/docker-entrypoint-initdb.d` | postgres | rw (init scripts) |
+| `./infra/trino/catalog/` | `/etc/trino/catalog` | trino | rw |
+| `./infra/hive/core-site.xml` | `/opt/hive/conf/core-site.xml` | hive-metastore | ro |
+| `./infra/hive/metastore-site.xml` | `/opt/hive/conf/metastore-site.xml` | hive-metastore | ro |
+
+## Host Port Mappings
+
+Services accessible from the host machine:
+
+| Host Port | Container | Container Port | Service |
+|-----------|-----------|----------------|---------|
+| 5432 | postgres | 5432 | PostgreSQL database |
+| 6379 | redis | 6379 | Redis cache and queues |
+| 9000 | minio | 9000 | MinIO S3 API |
+| 9001 | minio | 9001 | MinIO web console |
+| 11434 | ollama | 11434 | Ollama LLM API |
+| 8080 | trino | 8080 | Trino query engine |
+| 9083 | hive-metastore | 9083 | Hive Metastore thrift |
+| 8088 | superset | 8088 | Superset dashboard |
+| 8001 | symbol-registry | 8000 | Symbol Registry API |
+| 8002 | trading-engine | 8000 | Trading Engine API |
+| 8003 | risk-engine | 8000 | Risk Engine API |
+| 8004 | query-api | 8000 | Query API |
+| 3000 | dashboard | 8080 | React dashboard (nginx) |
+
+Services without host port mappings (internal only): scheduler, ingestion, parser, extractor, aggregation, recommendation, broker-adapter, lake-publisher, minio-init.
+
+## Environment Configuration
+
+### Shared Environment (`x-app-env` YAML anchor)
+
+All 13 application services and the scheduler receive these environment variables via the `x-app-env` anchor:
+
+| Variable | Value | Purpose |
+|----------|-------|---------|
+| `POSTGRES_HOST` | `postgres` | Docker Compose service name for PostgreSQL |
+| `POSTGRES_PORT` | `5432` | PostgreSQL port |
+| `POSTGRES_DB` | `stonks` | Database name |
+| `POSTGRES_USER` | `stonks` | Database user |
+| `POSTGRES_PASSWORD` | `stonks_dev` | Database password (dev default) |
+| `REDIS_HOST` | `redis` | Docker Compose service name for Redis |
+| `REDIS_PORT` | `6379` | Redis port |
+| `MINIO_ENDPOINT` | `minio:9000` | Docker Compose service name for MinIO |
+| `MINIO_ACCESS_KEY` | `minioadmin` | MinIO access key |
+| `MINIO_SECRET_KEY` | `minioadmin` | MinIO secret key |
+| `OLLAMA_BASE_URL` | `http://ollama:11434` | Docker Compose service name for Ollama |
+
+### `.env` File (API Keys)
+
+Three services load additional secrets from the `.env` file in the repository root via `env_file: .env`:
+
+| Variable | Required By | Purpose |
+|----------|-------------|---------|
+| `MARKET_DATA_API_KEY` | ingestion | Polygon.io market data API key |
+| `BROKER_API_KEY` | broker-adapter, trading-engine | Alpaca broker API key |
+| `BROKER_API_SECRET` | broker-adapter, trading-engine | Alpaca broker API secret |
+| `BROKER_BASE_URL` | broker-adapter, trading-engine | Alpaca API base URL (default: `https://paper-api.alpaca.markets`) |
+
+## Health Check Summary
+
+| Container | Health Check Command | Interval | Timeout | Retries | Start Period |
+|-----------|---------------------|----------|---------|---------|--------------|
+| postgres | `pg_isready -U stonks` | 5s | — | 5 | — |
+| redis | `redis-cli ping` | 5s | — | 5 | — |
+| minio | `mc ready local` | 5s | — | 5 | — |
+| symbol-registry | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
+| query-api | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
+| trading-engine | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
+| risk-engine | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
+| dashboard | `curl -f http://localhost:8080/` | 10s | 5s | 3 | 10s |
+| scheduler | `pgrep -f 'python -m services.scheduler.app'` | 10s | 5s | 3 | 15s |
+| ingestion | `pgrep -f 'python -m services.ingestion.worker'` | 10s | 5s | 3 | 15s |
+| parser | `pgrep -f 'python -m services.parser.worker'` | 10s | 5s | 3 | 15s |
+| extractor | `pgrep -f 'python -m services.extractor.main'` | 10s | 5s | 3 | 15s |
+| aggregation | `pgrep -f 'python -m services.aggregation.main'` | 10s | 5s | 3 | 15s |
+| recommendation | `pgrep -f 'python -m services.recommendation.main'` | 10s | 5s | 3 | 15s |
+| broker-adapter | `pgrep -f 'python -m services.adapters.broker_service'` | 10s | 5s | 3 | 15s |
+| lake-publisher | `pgrep -f 'python -m services.lake_publisher.jobs'` | 10s | 5s | 3 | 15s |
+
+Infrastructure services (ollama, trino, hive-metastore, superset) do not define health checks in docker-compose.yml. Application services that depend on ollama use `condition: service_started` instead of `condition: service_healthy`.
+
+## Internal Network Connectivity
+
+All containers share the default Docker Compose network. Services reference each other by their Compose service name as the hostname:
+
+| Hostname | Resolved To | Used By |
+|----------|-------------|---------|
+| `postgres` | PostgreSQL container | All 13 app services, superset |
+| `redis` | Redis container | scheduler, ingestion, parser, extractor, aggregation, recommendation, trading-engine, broker-adapter, query-api |
+| `minio` | MinIO container | ingestion, lake-publisher, query-api (via `minio:9000`) |
+| `ollama` | Ollama container | extractor (via `http://ollama:11434`) |
+| `hive-metastore` | Hive Metastore container | trino (thrift://hive-metastore:9083) |
+| `trino` | Trino container | superset (trino:8080) |
+| `query-api` | Query API container | dashboard (nginx proxy upstream) |
diff --git a/docs/architecture-kubernetes.md b/docs/architecture-kubernetes.md
new file mode 100644
index 0000000..475f312
--- /dev/null
+++ b/docs/architecture-kubernetes.md
@@ -0,0 +1,355 @@
+# Kubernetes Architecture — Stonks Oracle
+
+This document describes the Kubernetes deployment topology for Stonks Oracle, derived from the Helm chart at `infra/helm/stonks-oracle/`.
+
+All application workloads deploy to the `stonks-oracle` namespace. External cluster services (PostgreSQL, Redis, MinIO, Ollama) run in their own namespaces and are referenced via cross-namespace DNS.
+
+## Deployment Diagram
+
+```mermaid
+graph TB
+ %% ── External traffic ──────────────────────────────────────────
+ internet((Internet))
+
+ subgraph traefik ["kube-system (Traefik Ingress Controller)"]
+ direction LR
+ ing_dash["stonks.celestium.life"]
+ ing_api["stonks-api.celestium.life"]
+ ing_reg["stonks-registry.celestium.life"]
+ ing_trade["stonks-trading.celestium.life"]
+ ing_superset["stonks-dash.celestium.life"]
+ ing_trino["stonks-trino.celestium.life"]
+ end
+
+ internet --> traefik
+
+ %% ── stonks-oracle namespace ───────────────────────────────────
+ subgraph ns ["stonks-oracle namespace"]
+ direction TB
+
+ %% ── API Tier (ingress-facing) ─────────────────────────────
+ subgraph api_tier ["API Tier"]
+ direction LR
+ query_api["query-api
Deployment (1 replica)
:8000"]
+ symbol_registry["symbol-registry
Deployment (1 replica)
:8000"]
+ end
+
+ %% ── Frontend Tier ─────────────────────────────────────────
+ subgraph frontend_tier ["Frontend Tier"]
+ dashboard["dashboard
Deployment (1 replica)
:8080
nginx-unprivileged"]
+ end
+
+ %% ── Trading Tier ──────────────────────────────────────────
+ subgraph trading_tier ["Trading Tier"]
+ direction LR
+ trading_engine["trading-engine
Deployment (1 replica)
:8000"]
+ risk_engine["risk-engine
Deployment (1 replica)
:8000"]
+ broker_adapter["broker-adapter
Deployment (1 replica)
queue-driven worker"]
+ end
+
+ %% ── Orchestration Tier ────────────────────────────────────
+ subgraph orchestration_tier ["Orchestration Tier"]
+ scheduler["scheduler
Deployment (1 replica)
runs migrations + seed"]
+ end
+
+ %% ── Processing Tier (pipeline workers) ────────────────────
+ subgraph processing_tier ["Processing Tier (pipeline workers)"]
+ direction LR
+ ingestion["ingestion
Deployment (2 replicas)"]
+ parser["parser
Deployment (2 replicas)"]
+ extractor["extractor
Deployment (1 replica)"]
+ aggregation["aggregation
Deployment (4 replicas)"]
+ recommendation["recommendation
Deployment (1 replica)"]
+ end
+
+ %% ── Analytics Tier ────────────────────────────────────────
+ subgraph analytics_tier ["Analytics Tier"]
+ direction LR
+ lake_publisher["lake-publisher
Deployment (1 replica)
queue-driven worker"]
+ hive_metastore["hive-metastore
Deployment (1 replica)
:9083
apache/hive:4.0.0"]
+ trino["trino
Deployment (1 replica)
:8080
trinodb/trino:latest"]
+ superset["superset
Deployment (1 replica)
:8088
custom image"]
+ end
+
+ %% ── Helm Secrets ──────────────────────────────────────────
+ subgraph secrets_block ["Helm-Managed Secrets"]
+ direction LR
+ sec_core["stonks-core-secrets
POSTGRES_PASSWORD
MINIO_ACCESS_KEY
MINIO_SECRET_KEY
REDIS_PASSWORD"]
+ sec_broker["stonks-broker-secrets
BROKER_API_KEY
BROKER_API_SECRET
BROKER_BASE_URL"]
+ sec_market["stonks-market-secrets
MARKET_DATA_API_KEY"]
+ sec_gmail["stonks-gmail-secrets
GMAIL_SENDER
GMAIL_RECIPIENT
GMAIL_APP_PASSWORD"]
+ sec_dashboard["stonks-dashboard-secrets
SUPERSET_SECRET_KEY
SUPERSET_ADMIN_PASSWORD"]
+ end
+
+ %% ── ConfigMap ─────────────────────────────────────────────
+ configmap["stonks-config
ConfigMap
All env vars from values.yaml config block"]
+ end
+
+ %% ── External Cluster Services ─────────────────────────────────
+ subgraph pg_ns ["postgresql-service namespace"]
+ postgres[("PostgreSQL
postgresql-rw:5432")]
+ end
+
+ subgraph redis_ns ["redis-service namespace"]
+ redis[("Redis
redis-master:6379")]
+ end
+
+ subgraph minio_ns ["minio-service namespace"]
+ minio[("MinIO
minio:80")]
+ end
+
+ subgraph ollama_ns ["ollama-service namespace"]
+ ollama[("Ollama
ollama:11434
GPU: 4070 Ti Super")]
+ end
+
+ %% ── Ingress Routes ────────────────────────────────────────────
+ ing_dash -->|":8080"| dashboard
+ ing_api -->|":8000"| query_api
+ ing_reg -->|":8000"| symbol_registry
+ ing_trade -->|":8000"| trading_engine
+ ing_superset -->|":8088"| superset
+ ing_trino -->|":8080"| trino
+
+ %% ── Dashboard → Backend APIs ──────────────────────────────────
+ dashboard -.->|"/api/ proxy"| query_api
+ dashboard -.->|"/registry/ proxy"| symbol_registry
+ dashboard -.->|"/risk/ proxy"| risk_engine
+
+ %% ── Pipeline data flow (via Redis queues) ─────────────────────
+ scheduler -->|"enqueue jobs"| redis
+ ingestion -->|"stonks:queue:parsing"| redis
+ parser -->|"stonks:queue:extraction"| redis
+ extractor -->|"stonks:queue:aggregation"| redis
+ aggregation -->|"stonks:queue:recommendation"| redis
+ recommendation -->|"stonks:queue:trading_decisions"| redis
+ trading_engine -->|"stonks:queue:broker_orders"| redis
+ broker_adapter -->|"read orders"| redis
+ lake_publisher -->|"stonks:queue:lake_publish"| redis
+
+ %% ── External service connections ──────────────────────────────
+ scheduler --> postgres
+ scheduler --> redis
+ ingestion --> postgres
+ ingestion --> redis
+ ingestion --> minio
+ parser --> postgres
+ parser --> redis
+ extractor --> postgres
+ extractor --> redis
+ extractor --> ollama
+ aggregation --> postgres
+ aggregation --> redis
+ recommendation --> postgres
+ recommendation --> redis
+ trading_engine --> postgres
+ trading_engine --> redis
+ risk_engine --> postgres
+ broker_adapter --> postgres
+ broker_adapter --> redis
+ lake_publisher --> postgres
+ lake_publisher --> minio
+ query_api --> postgres
+ query_api --> redis
+ query_api --> minio
+ symbol_registry --> postgres
+
+ %% ── Analytics plane connections ───────────────────────────────
+ lake_publisher -->|"Parquet → s3a://stonks-lakehouse"| minio
+ hive_metastore -->|"s3a:// catalog"| minio
+ trino -->|"thrift://hive-metastore:9083"| hive_metastore
+ superset -->|"trino:8080"| trino
+ query_api -->|"trino:8080"| trino
+ superset --> postgres
+ superset --> redis
+
+ %% ── Trading tier external egress ──────────────────────────────
+ trading_engine -->|"HTTPS :443
Alpaca API"| internet
+ trading_engine -->|"SMTP :587
Gmail notifications"| internet
+ broker_adapter -->|"HTTPS :443
Alpaca API"| internet
+ ingestion -->|"HTTPS :443
Polygon.io / News APIs"| internet
+
+ %% ── Secret consumption ────────────────────────────────────────
+ sec_core -.-> query_api
+ sec_core -.-> symbol_registry
+ sec_core -.-> scheduler
+ sec_core -.-> ingestion
+ sec_core -.-> parser
+ sec_core -.-> extractor
+ sec_core -.-> aggregation
+ sec_core -.-> recommendation
+ sec_core -.-> trading_engine
+ sec_core -.-> risk_engine
+ sec_core -.-> broker_adapter
+ sec_core -.-> lake_publisher
+ sec_core -.-> hive_metastore
+ sec_core -.-> trino
+ sec_core -.-> superset
+
+ sec_broker -.-> ingestion
+ sec_broker -.-> trading_engine
+ sec_broker -.-> risk_engine
+ sec_broker -.-> broker_adapter
+
+ sec_market -.-> ingestion
+
+ sec_gmail -.-> trading_engine
+
+ sec_dashboard -.-> superset
+
+ configmap -.-> query_api
+ configmap -.-> symbol_registry
+ configmap -.-> scheduler
+ configmap -.-> ingestion
+ configmap -.-> parser
+ configmap -.-> extractor
+ configmap -.-> aggregation
+ configmap -.-> recommendation
+ configmap -.-> trading_engine
+ configmap -.-> risk_engine
+ configmap -.-> broker_adapter
+ configmap -.-> lake_publisher
+ configmap -.-> superset
+
+ %% ── Styles ────────────────────────────────────────────────────
+ classDef apiSvc fill:#4a90d9,stroke:#2c5f8a,color:#fff
+ classDef frontendSvc fill:#50c878,stroke:#2e7d46,color:#fff
+ classDef tradingSvc fill:#e8a838,stroke:#b07d1a,color:#fff
+ classDef processSvc fill:#9b59b6,stroke:#6c3483,color:#fff
+ classDef orchSvc fill:#1abc9c,stroke:#148f77,color:#fff
+ classDef analyticsSvc fill:#e74c3c,stroke:#a93226,color:#fff
+ classDef extSvc fill:#95a5a6,stroke:#717d7e,color:#fff
+ classDef secretSvc fill:#f5f5dc,stroke:#999,color:#333
+ classDef configSvc fill:#dfe6e9,stroke:#999,color:#333
+
+ class query_api,symbol_registry apiSvc
+ class dashboard frontendSvc
+ class trading_engine,risk_engine,broker_adapter tradingSvc
+ class scheduler orchSvc
+ class ingestion,parser,extractor,aggregation,recommendation processSvc
+ class lake_publisher,hive_metastore,trino,superset analyticsSvc
+ class postgres,redis,minio,ollama extSvc
+ class sec_core,sec_broker,sec_market,sec_gmail,sec_dashboard secretSvc
+ class configmap configSvc
+```
+
+## Network Policy Boundaries
+
+The Helm chart deploys a **default-deny-ingress** policy that blocks all inbound traffic to pods in the `stonks-oracle` namespace. Each service that needs inbound connections has an explicit allow policy:
+
+```mermaid
+graph LR
+ subgraph netpol ["Network Policies — stonks-oracle namespace"]
+ direction TB
+
+ deny["🔒 default-deny-ingress
Blocks ALL ingress to all pods"]
+
+ subgraph allows ["Explicit Allow Rules"]
+ direction TB
+
+ np_dash["allow-dashboard-ingress
dashboard :8080
← kube-system (Traefik)"]
+
+ np_api["allow-query-api-ingress
query-api :8000
← kube-system (Traefik)
← dashboard pod"]
+
+ np_reg["allow-symbol-registry-ingress
symbol-registry :8000
← kube-system (Traefik)
← dashboard pod"]
+
+ np_trade["allow-trading-engine-ingress
trading-engine :8000
← kube-system (Traefik)
← query-api pod
← dashboard pod
Egress: PostgreSQL :5432,
Redis :6379, HTTPS :443, SMTP :587"]
+
+ np_risk["allow-risk-engine-ingress
risk-engine :8000
← broker-adapter pod
← query-api pod
← dashboard pod"]
+
+ np_superset["allow-superset-ingress
superset :8088
← kube-system (Traefik)"]
+
+ np_trino["allow-trino-ingress
trino :8080
← superset pod
← query-api pod
← kube-system (Traefik)"]
+
+ np_hive["allow-hive-metastore-ingress
hive-metastore :9083
← trino pod
← lake-publisher pod"]
+
+ np_broker["deny-broker-adapter-ingress
broker-adapter
No inbound traffic allowed"]
+ end
+ end
+
+ style deny fill:#e74c3c,stroke:#c0392b,color:#fff
+ style np_broker fill:#e74c3c,stroke:#c0392b,color:#fff
+ style np_dash fill:#2ecc71,stroke:#27ae60,color:#fff
+ style np_api fill:#2ecc71,stroke:#27ae60,color:#fff
+ style np_reg fill:#2ecc71,stroke:#27ae60,color:#fff
+ style np_trade fill:#f39c12,stroke:#d68910,color:#fff
+ style np_risk fill:#f39c12,stroke:#d68910,color:#fff
+ style np_superset fill:#2ecc71,stroke:#27ae60,color:#fff
+ style np_trino fill:#2ecc71,stroke:#27ae60,color:#fff
+ style np_hive fill:#3498db,stroke:#2980b9,color:#fff
+```
+
+### Services Without Ingress Policies (Pipeline Workers)
+
+The following services have **no inbound network policy** — they are queue-driven workers that only make outbound connections to PostgreSQL, Redis, MinIO, and Ollama. The default-deny-ingress policy blocks any unsolicited inbound traffic:
+
+| Service | Tier | Behavior |
+|---------|------|----------|
+| scheduler | orchestration | Polls DB, enqueues to Redis |
+| ingestion | processing | Reads from `stonks:queue:ingestion`, writes to DB/MinIO/Redis |
+| parser | processing | Reads from `stonks:queue:parsing`, writes to DB/Redis |
+| extractor | processing | Reads from `stonks:queue:extraction`, calls Ollama, writes to DB/Redis |
+| aggregation | processing | Reads from `stonks:queue:aggregation`, writes to DB/Redis |
+| recommendation | processing | Reads from `stonks:queue:recommendation`, writes to DB/Redis |
+| lake-publisher | analytics | Reads from `stonks:queue:lake_publish`, writes Parquet to MinIO |
+
+## Service Tier Summary
+
+| Tier | Services | Ingress? | Replicas | Notes |
+|------|----------|----------|----------|-------|
+| **api** | query-api, symbol-registry | Yes (Traefik) | 1 each | FastAPI, readiness probes on `/docs` |
+| **frontend** | dashboard | Yes (Traefik) | 1 | nginx-unprivileged on :8080, proxies to API services |
+| **trading** | trading-engine, risk-engine, broker-adapter | trading-engine: Yes; risk-engine: internal only; broker-adapter: denied | 1 each | trading-engine has egress to Alpaca + Gmail |
+| **orchestration** | scheduler | No | 1 | Runs DB migrations + seed as init containers |
+| **processing** | ingestion, parser, extractor, aggregation, recommendation | No | 2, 2, 1, 4, 1 | Pipeline-gated by `pipelineEnabled` toggle |
+| **analytics** | lake-publisher, trino, hive-metastore, superset | trino + superset: Yes; others: No | 1 each | lake-publisher is pipeline-gated |
+
+## Secret Consumption Map
+
+| Secret | Keys | Consumers |
+|--------|------|-----------|
+| `stonks-core-secrets` | POSTGRES_PASSWORD, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, REDIS_PASSWORD | All 13 app services + hive-metastore, trino, superset |
+| `stonks-broker-secrets` | BROKER_API_KEY, BROKER_API_SECRET, BROKER_BASE_URL | ingestion, trading-engine, risk-engine, broker-adapter |
+| `stonks-market-secrets` | MARKET_DATA_API_KEY | ingestion |
+| `stonks-gmail-secrets` | GMAIL_SENDER, GMAIL_RECIPIENT, GMAIL_APP_PASSWORD | trading-engine |
+| `stonks-dashboard-secrets` | SUPERSET_SECRET_KEY, SUPERSET_ADMIN_PASSWORD | superset |
+
+## Pipeline Toggle
+
+Setting `pipelineEnabled: false` in `values.yaml` scales all services with `pipeline: true` to 0 replicas. This affects:
+
+- scheduler, ingestion, parser, extractor, aggregation, recommendation, broker-adapter, lake-publisher
+
+API-tier services (query-api, symbol-registry), trading-tier services (trading-engine, risk-engine), analytics services (trino, hive-metastore, superset), and the dashboard always run regardless of this toggle.
+
+## External Cluster Services
+
+These services run outside the `stonks-oracle` namespace and are referenced via cross-namespace DNS:
+
+| Service | Namespace | DNS | Port | Notes |
+|---------|-----------|-----|------|-------|
+| PostgreSQL | `postgresql-service` | `postgresql-rw.postgresql-service.svc.cluster.local` | 5432 | CloudNativePG managed |
+| Redis | `redis-service` | `redis-master.redis-service.svc.cluster.local` | 6379 | Password in `stonks-core-secrets` |
+| MinIO | `minio-service` | `minio.minio-service.svc.cluster.local` | 80 | S3-compatible object store |
+| Ollama | `ollama-service` | `ollama.ollama-service.svc.cluster.local` | 11434 | LLM inference, GPU: 4070 Ti Super 16GB |
+
+## Analytics Plane
+
+The analytics stack runs within the `stonks-oracle` namespace:
+
+1. **Lake Publisher** writes Parquet fact tables to MinIO at `s3a://stonks-lakehouse/warehouse`
+2. **Hive Metastore** (Apache Hive 4.0.0) manages table metadata, backed by embedded Derby DB with a PVC for persistence. Connects to MinIO for S3A filesystem access.
+3. **Trino** queries the lakehouse via Hive Metastore (thrift://hive-metastore:9083). Exposes two catalogs: `lakehouse` (Hive connector) and `iceberg` (Iceberg connector). Both connect to MinIO for data access.
+4. **Superset** connects to Trino for lakehouse queries and to PostgreSQL for its metadata DB. Uses Redis for caching. Exposed externally via Traefik ingress.
+
+## Ingress Routes
+
+All ingress resources use the `traefik` IngressClass with TLS certificates issued by the `ca-issuer` ClusterIssuer:
+
+| Domain | Backend Service | Port | TLS Secret |
+|--------|----------------|------|------------|
+| `stonks.celestium.life` | dashboard | 8080 | `stonks-dashboard-tls` |
+| `stonks-api.celestium.life` | query-api | 8000 | `stonks-api-tls` |
+| `stonks-registry.celestium.life` | symbol-registry | 8000 | `stonks-registry-tls` |
+| `stonks-trading.celestium.life` | trading-engine | 8000 | `stonks-trading-tls` |
+| `stonks-dash.celestium.life` | superset | 8088 | `stonks-dash-tls` |
+| `stonks-trino.celestium.life` | trino | 8080 | `stonks-trino-tls` |
diff --git a/docs/backup-restore.md b/docs/backup-restore.md
new file mode 100644
index 0000000..658ff93
--- /dev/null
+++ b/docs/backup-restore.md
@@ -0,0 +1,440 @@
+# Backup and Restore Guide
+
+This guide documents every backup and restore script in the Stonks Oracle platform, their CLI options, storage locations, retention policies, and procedures for disaster recovery.
+
+## Overview
+
+Stonks Oracle provides two tiers of backup tooling:
+
+| Tier | Scripts | Scope | Storage |
+|------|---------|-------|---------|
+| **Local (kubectl-based)** | `backup-db.sh`, `restore-db.sh`, `backup-redis.sh` | Individual data stores, streamed to the operator's machine | `~/backups/stonks-oracle/` (local filesystem) |
+| **Cluster (Kubernetes Job)** | `backup.sh`, `restore.sh` | Full platform (PostgreSQL + all MinIO buckets) | NFS share at `192.168.42.8:/volume1/Kubernetes/stonks` |
+
+All scripts live in the `scripts/` directory and require `kubectl` access to the cluster.
+
+---
+
+## Local Backup Scripts
+
+### `backup-db.sh` — PostgreSQL Database Backup
+
+Creates a compressed `pg_dump` of the `stonks` database and optionally uploads it to MinIO.
+
+**Usage:**
+
+```bash
+./scripts/backup-db.sh # backup to local file
+./scripts/backup-db.sh --upload-minio # backup + upload to MinIO
+```
+
+**CLI Arguments:**
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `--upload-minio` | No | Upload the backup file to the `stonks-backups` MinIO bucket after creating it |
+
+**Environment Variables:**
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BACKUP_DIR` | `~/backups/stonks-oracle` | Local directory where backup files are stored |
+
+**What it captures:**
+
+- Full `pg_dump` of the `stonks` database (all tables, data, sequences)
+- Dump flags: `--no-owner --no-privileges --clean --if-exists`
+- Output format: gzip-compressed SQL (`.sql.gz`)
+
+**How it works:**
+
+1. Runs `pg_dump` inside the PostgreSQL pod (`postgresql-1` in `postgresql-service` namespace) and streams the compressed output to the local machine
+2. Validates the backup is non-empty and counts tables as a sanity check
+3. If `--upload-minio` is specified, attempts to create the `stonks-backups` bucket (if it doesn't exist) and stages the file for upload
+4. Prunes old backups, keeping only the last 7 files matching `stonks-*.sql.gz`
+
+**Storage:**
+
+- Local path: `~/backups/stonks-oracle/stonks-.sql.gz`
+- MinIO bucket (optional): `stonks-backups`
+
+**Retention:** Keeps the last 7 backups. Older files matching `stonks-*.sql.gz` in the backup directory are automatically deleted.
+
+---
+
+### `backup-redis.sh` — Redis State Backup
+
+Triggers a Redis `BGSAVE` and copies the RDB dump file to the local machine.
+
+**Usage:**
+
+```bash
+./scripts/backup-redis.sh
+```
+
+**CLI Arguments:** None.
+
+**Environment Variables:**
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BACKUP_DIR` | `~/backups/stonks-oracle` | Local directory where the RDB file is stored |
+| `REDIS_PASSWORD` | `PSCh4ng3me!` | Redis authentication password |
+
+**What it captures:**
+
+- Redis RDB snapshot (`dump.rdb`) containing all in-memory state: deduplication markers, queue contents, rate-limit counters, cached values
+
+**How it works:**
+
+1. Triggers `BGSAVE` on the Redis master pod (`redis-master-0` in `redis-service` namespace)
+2. Waits 5 seconds for the background save to complete, then logs the `LASTSAVE` timestamp
+3. Copies the RDB file from the pod. Tries `/data/dump.rdb` first, then falls back to `/var/lib/redis/dump.rdb` and `/bitnami/redis/data/dump.rdb`
+4. Prints Redis keyspace statistics for verification
+
+**Storage:**
+
+- Local path: `~/backups/stonks-oracle/redis-.rdb`
+
+**Retention:** No automatic pruning. Old Redis backups accumulate and must be cleaned up manually.
+
+---
+
+### `restore-db.sh` — PostgreSQL Database Restore
+
+Restores a `pg_dump` backup into the `stonks` database with full service scale-down/scale-up.
+
+**Usage:**
+
+```bash
+./scripts/restore-db.sh
+./scripts/restore-db.sh ~/backups/stonks-oracle/stonks-20260415-180000.sql.gz
+```
+
+If called without arguments, lists available backups in `~/backups/stonks-oracle/`.
+
+**CLI Arguments:**
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `` | Yes | Path to the gzip-compressed SQL backup file to restore |
+
+**What it restores:**
+
+- All tables, data, sequences, and indexes in the `stonks` database
+- Re-grants `ALL PRIVILEGES` to the `stonks` user on all tables and sequences after restore
+
+**Service scale-down/scale-up procedure:**
+
+1. **Terminates active connections** — Runs `pg_terminate_backend()` for all connections to the `stonks` database
+2. **Scales down all deployments** in the `stonks-oracle` namespace to 0 replicas to prevent reconnections
+3. **Waits 10 seconds** for pods to terminate
+4. **Restores the backup** using `psql --single-transaction` (piped from `zcat`)
+5. **Re-grants permissions** to the `stonks` user
+6. **Verifies** the restore by counting tables
+7. **Scales all deployments back to 1 replica**, then scales `ingestion` and `parser` to 2 replicas
+
+**Data loss implications:**
+
+> **WARNING:** This replaces ALL data in the `stonks` database with the backup contents. Any data written after the backup was taken is permanently lost. The script requires interactive confirmation — you must type `yes` to proceed.
+
+---
+
+## Cluster Backup Scripts (Kubernetes Jobs)
+
+### `backup.sh` — Full Platform Backup (PostgreSQL + MinIO)
+
+Runs a Kubernetes Job that backs up both PostgreSQL and all MinIO buckets to an NFS share.
+
+**Usage:**
+
+```bash
+bash scripts/backup.sh
+```
+
+**CLI Arguments:** None.
+
+**What it captures:**
+
+- **PostgreSQL**: Full `pg_dump` in custom format (`-Fc`) as `stonks.pgdump`
+- **MinIO buckets** (8 buckets mirrored):
+ - `stonks-raw-market` — Raw market data from Polygon.io
+ - `stonks-raw-news` — Raw news articles
+ - `stonks-raw-filings` — Raw SEC filings
+ - `stonks-normalized` — Normalized documents
+ - `stonks-llm-prompts` — LLM prompt logs
+ - `stonks-llm-results` — LLM extraction results
+ - `stonks-lakehouse` — Parquet fact tables for Trino
+ - `stonks-audit` — Audit trail artifacts
+- **Manifest**: `manifest.json` with backup name, timestamp, and bucket list
+
+**How it works:**
+
+1. Deletes any previous `stonks-backup` Job
+2. Creates a Kubernetes Job using `postgres:18-alpine` with NFS volume mount and MinIO credentials from cluster secrets
+3. Inside the Job container:
+ - Runs `pg_dump` with credentials from `stonks-config` ConfigMap and `stonks-core-secrets` Secret
+ - Installs the MinIO client (`mc`) and mirrors each bucket to the NFS backup directory
+ - Writes a `manifest.json` and updates the `latest` symlink
+4. Waits up to 600 seconds (10 minutes) for the Job to complete
+5. Job auto-cleans after 300 seconds (`ttlSecondsAfterFinished`)
+
+**Storage:**
+
+- NFS path: `192.168.42.8:/volume1/Kubernetes/stonks//`
+- Directory structure:
+ ```
+ stonks-backup-YYYYMMDD-HHMMSS/
+ ├── stonks.pgdump # PostgreSQL custom-format dump
+ ├── manifest.json # Backup metadata
+ └── minio/
+ ├── stonks-raw-market/ # Mirrored bucket contents
+ ├── stonks-raw-news/
+ ├── stonks-raw-filings/
+ ├── stonks-normalized/
+ ├── stonks-llm-prompts/
+ ├── stonks-llm-results/
+ ├── stonks-lakehouse/
+ └── stonks-audit/
+ ```
+- A `latest` symlink always points to the most recent backup
+
+**Retention:** No automatic pruning on NFS. Old backups must be cleaned up manually.
+
+---
+
+### `restore.sh` — Full Platform Restore (PostgreSQL + MinIO)
+
+Runs a Kubernetes Job that restores both PostgreSQL and MinIO buckets from an NFS backup.
+
+**Usage:**
+
+```bash
+bash scripts/restore.sh # restore from "latest" symlink
+bash scripts/restore.sh # restore a specific backup
+```
+
+**CLI Arguments:**
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `` | No | Name of the backup directory on NFS. Defaults to `latest` (symlink to most recent backup) |
+
+**What it restores:**
+
+- **PostgreSQL**: Full database restore using `pg_restore --clean --if-exists --no-owner --no-acl`
+- **MinIO buckets**: All 8 buckets mirrored back with `mc mirror --overwrite`
+
+**How it works:**
+
+1. Prints a warning and gives 5 seconds to abort (Ctrl+C)
+2. Deletes any previous `stonks-restore` Job
+3. Creates a Kubernetes Job that:
+ - Validates the backup exists (`stonks.pgdump` file present)
+ - Restores PostgreSQL using `pg_restore` with `--clean` (drops and recreates objects)
+ - Installs `mc` and mirrors each bucket back from NFS to MinIO
+ - Verifies the restore by querying row counts for key tables (companies, documents, intelligence, impacts, trends, recommendations)
+4. Waits up to 600 seconds for the Job to complete
+
+**Data loss implications:**
+
+> **WARNING:** This will DROP and recreate all objects in the `stonks` database. All MinIO bucket contents are overwritten. Any data written after the backup was taken is permanently lost. The script provides a 5-second abort window before proceeding.
+
+**Post-restore steps:**
+
+After the restore completes, restart all services to pick up the restored state:
+
+```bash
+kubectl rollout restart deployment -n stonks-oracle --all
+```
+
+---
+
+## MinIO Upload Option (`--upload-minio`)
+
+The `backup-db.sh` script supports `--upload-minio` for off-host storage of database backups. When enabled:
+
+1. The script connects to MinIO through an ingestion pod in the `stonks-oracle` namespace
+2. Creates the `stonks-backups` bucket if it doesn't already exist
+3. Stages the backup file for upload
+
+This provides a second copy of the database backup on object storage, separate from the operator's local filesystem. The full cluster backup (`backup.sh`) stores backups on NFS and does not use this flag — it backs up MinIO bucket *contents* rather than uploading database dumps *to* MinIO.
+
+---
+
+## Full Nuke and Rebuild Procedure
+
+When a complete platform reset is needed (corrupted state, major schema changes, fresh start), follow this procedure:
+
+### Step 1: Tear Down Services
+
+```bash
+bash ~/sources/kube/stonks-oracle/runmelast.sh
+```
+
+This runs from `gremlin-1` and performs a Helm uninstall, cleaning up all Kubernetes resources in the `stonks-oracle` namespace. Database, MinIO, and Redis data are preserved (they run in separate namespaces).
+
+### Step 2: Terminate Database Connections
+
+```bash
+kubectl exec -n postgresql-service postgresql-1 -c postgres -- \
+ psql -U postgres -c \
+ "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'stonks' AND pid <> pg_backend_pid();"
+```
+
+### Step 3: Drop the Database
+
+```bash
+kubectl exec -n postgresql-service postgresql-1 -c postgres -- \
+ psql -U postgres -c "DROP DATABASE IF EXISTS stonks;"
+```
+
+### Step 4: Flush Redis
+
+Clear all `stonks:*` keys to reset deduplication markers, queue contents, and cached state:
+
+```bash
+kubectl exec -n redis-service redis-master-0 -- \
+ redis-cli -a 'PSCh4ng3me!' --scan --pattern 'stonks:*' | \
+ xargs -L 100 kubectl exec -n redis-service redis-master-0 -- \
+ redis-cli -a 'PSCh4ng3me!' DEL
+```
+
+### Step 5: Redeploy
+
+```bash
+bash ~/sources/kube/stonks-oracle/runmefirst.sh
+```
+
+This runs from `gremlin-1` and performs:
+- Database creation and migration (all `infra/migrations/*.sql` files applied in order)
+- Helm install with secrets injected via `--set` flags
+- Rolling restart of all deployments
+
+### Step 6: Re-seed the Symbol Registry
+
+```bash
+POSTGRES_HOST=postgresql-rw.postgresql-service.svc.cluster.local \
+POSTGRES_PASSWORD='St0nks0racl3!' \
+POSTGRES_USER=stonks \
+POSTGRES_DB=stonks \
+.venv/bin/python -m services.symbol_registry.seed
+```
+
+This populates the 50 tracked companies across 10 sectors and 46 competitor relationships.
+
+---
+
+## Recommended Backup Schedules
+
+### Daily Database Backup (cron)
+
+Run `backup-db.sh` daily on a machine with `kubectl` access. The built-in retention keeps the last 7 backups automatically.
+
+```cron
+# Daily database backup at 2:00 AM
+0 2 * * * /path/to/stonks-oracle/scripts/backup-db.sh --upload-minio >> /var/log/stonks-backup.log 2>&1
+```
+
+### Weekly Full Backup (cron)
+
+Run the full cluster backup weekly to capture both PostgreSQL and MinIO data on NFS:
+
+```cron
+# Weekly full backup (PostgreSQL + MinIO) on Sundays at 3:00 AM
+0 3 * * 0 /path/to/stonks-oracle/scripts/backup.sh >> /var/log/stonks-full-backup.log 2>&1
+```
+
+### Redis Backup Before Deployments
+
+Redis state is transient (queues, dedup markers, caches) and rebuilds naturally. Back up Redis before major deployments or database resets as a precaution:
+
+```bash
+./scripts/backup-redis.sh
+```
+
+### Kubernetes CronJobs
+
+For fully automated in-cluster backups, create a CronJob based on the same Job spec used by `backup.sh`:
+
+```yaml
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+ name: stonks-backup
+ namespace: stonks-oracle
+spec:
+ schedule: "0 2 * * *" # Daily at 2:00 AM UTC
+ concurrencyPolicy: Forbid
+ successfulJobsHistoryLimit: 3
+ failedJobsHistoryLimit: 3
+ jobTemplate:
+ spec:
+ ttlSecondsAfterFinished: 3600
+ backoffLimit: 1
+ template:
+ spec:
+ restartPolicy: Never
+ volumes:
+ - name: nfs-backup
+ nfs:
+ server: 192.168.42.8
+ path: /volume1/Kubernetes/stonks
+ containers:
+ - name: backup
+ image: postgres:18-alpine
+ volumeMounts:
+ - name: nfs-backup
+ mountPath: /backup
+ envFrom:
+ - configMapRef:
+ name: stonks-config
+ - secretRef:
+ name: stonks-core-secrets
+ env:
+ - name: MINIO_ACCESS_KEY
+ valueFrom:
+ secretKeyRef:
+ name: stonks-core-secrets
+ key: MINIO_ACCESS_KEY
+ - name: MINIO_SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: stonks-core-secrets
+ key: MINIO_SECRET_KEY
+ command: ["sh", "-c"]
+ args:
+ - |
+ set -e
+ apk add --no-cache curl ca-certificates
+ STAMP="stonks-backup-$(date +%Y%m%d-%H%M%S)"
+ DIR="/backup/${STAMP}"
+ mkdir -p "${DIR}/minio"
+
+ # PostgreSQL backup
+ PGPASSWORD="${POSTGRES_PASSWORD}" pg_dump \
+ -h "${POSTGRES_HOST}" -p "${POSTGRES_PORT}" \
+ -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" \
+ --no-owner --no-acl -Fc \
+ -f "${DIR}/stonks.pgdump"
+
+ # MinIO backup
+ curl -sL https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc
+ chmod +x /usr/local/bin/mc
+ mc alias set backup "http://${MINIO_ENDPOINT}" "${MINIO_ACCESS_KEY}" "${MINIO_SECRET_KEY}" --api S3v4
+
+ for bucket in stonks-raw-market stonks-raw-news stonks-raw-filings stonks-normalized stonks-llm-prompts stonks-llm-results stonks-lakehouse stonks-audit; do
+ mc mirror "backup/${bucket}" "${DIR}/minio/${bucket}/" 2>/dev/null || true
+ done
+
+ ln -sfn "${STAMP}" /backup/latest
+ echo "Backup complete: ${DIR}"
+```
+
+### Recommended Schedule Summary
+
+| What | Frequency | Script | Retention |
+|------|-----------|--------|-----------|
+| Database only | Daily | `backup-db.sh --upload-minio` | Last 7 (auto-pruned) |
+| Full platform (DB + MinIO) | Weekly | `backup.sh` | Manual cleanup on NFS |
+| Redis snapshot | Before deployments | `backup-redis.sh` | Manual cleanup |
diff --git a/docs/docker-deployment.md b/docs/docker-deployment.md
new file mode 100644
index 0000000..beddb66
--- /dev/null
+++ b/docs/docker-deployment.md
@@ -0,0 +1,627 @@
+# Docker Deployment Guide
+
+This guide covers running the full Stonks Oracle platform locally using Docker Compose. It documents every service, environment variable, volume mount, health check, and operational command.
+
+## Prerequisites
+
+- Docker Engine 24+ and Docker Compose v2
+- At least 16 GB RAM (Ollama + Trino + all services)
+- API keys for Polygon.io and Alpaca (optional — platform runs in degraded mode without them)
+
+## Quick Start
+
+```bash
+# 1. Clone the repository
+git clone && cd stonks-oracle
+
+# 2. Configure API keys
+cp .env.example .env # or edit the existing .env
+# Fill in MARKET_DATA_API_KEY, BROKER_API_KEY, BROKER_API_SECRET
+
+# 3. Start everything
+docker compose up -d
+
+# 4. Verify all services are healthy
+docker compose ps
+
+# 5. Access the dashboard
+open http://localhost:3000
+```
+
+---
+
+## Service Inventory
+
+### Infrastructure Services
+
+| Service | Image | Ports | Volumes | Purpose |
+|---------|-------|-------|---------|---------|
+| `postgres` | `postgres:16-alpine` | `5432:5432` | `pgdata` → `/var/lib/postgresql/data`, `./infra/migrations` → `/docker-entrypoint-initdb.d` | Primary database; migrations auto-applied on first start |
+| `redis` | `redis:7-alpine` | `6379:6379` | — | Queue broker, caching, deduplication |
+| `minio` | `minio/minio:latest` | `9000:9000` (API), `9001:9001` (console) | `miniodata` → `/data` | Object storage for raw artifacts and lakehouse |
+| `minio-init` | `minio/mc:latest` | — | — | One-shot init container that creates required buckets |
+| `ollama` | `ollama/ollama:latest` | `11434:11434` | `ollama_models` → `/root/.ollama` | LLM inference server for extraction and classification |
+| `trino` | `trinodb/trino:latest` | `8080:8080` | `./infra/trino/catalog` → `/etc/trino/catalog` | SQL query engine over the lakehouse |
+| `hive-metastore` | `apache/hive:4.0.0` | `9083:9083` | `hive_data` → `/opt/hive/data`, `./infra/hive/core-site.xml` → `/opt/hive/conf/core-site.xml`, `./infra/hive/metastore-site.xml` → `/opt/hive/conf/metastore-site.xml` | Iceberg/Hive metadata catalog for Trino |
+| `superset` | `apache/superset:latest` | `8088:8088` | `superset_data` → `/app/superset_home` | BI dashboards over Trino |
+
+### Application Services
+
+| Service | Dockerfile | `SERVICE_CMD` / Command | Ports | Depends On |
+|---------|-----------|------------------------|-------|------------|
+| `scheduler` | `docker/Dockerfile.scheduler` | `python -m services.scheduler.app` | — | postgres (healthy), redis (healthy) |
+| `symbol-registry` | `docker/Dockerfile` | `uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000` | `8001:8000` | postgres (healthy) |
+| `ingestion` | `docker/Dockerfile` | `python -m services.ingestion.worker` | — | postgres (healthy), redis (healthy), minio (healthy) |
+| `parser` | `docker/Dockerfile` | `python -m services.parser.worker` | — | postgres (healthy), redis (healthy) |
+| `extractor` | `docker/Dockerfile` | `python -m services.extractor.main` | — | postgres (healthy), redis (healthy), ollama (started) |
+| `aggregation` | `docker/Dockerfile` | `python -m services.aggregation.main` | — | postgres (healthy), redis (healthy) |
+| `recommendation` | `docker/Dockerfile` | `python -m services.recommendation.main` | — | postgres (healthy), redis (healthy) |
+| `trading-engine` | `docker/Dockerfile` | `uvicorn services.trading.app:app --host 0.0.0.0 --port 8000` | `8002:8000` | postgres (healthy), redis (healthy) |
+| `risk-engine` | `docker/Dockerfile` | `uvicorn services.risk.app:app --host 0.0.0.0 --port 8000` | `8003:8000` | postgres (healthy) |
+| `broker-adapter` | `docker/Dockerfile` | `python -m services.adapters.broker_service` | — | postgres (healthy), redis (healthy) |
+| `lake-publisher` | `docker/Dockerfile` | `python -m services.lake_publisher.jobs` | — | postgres (healthy), minio (healthy) |
+| `query-api` | `docker/Dockerfile` | `uvicorn services.api.app:app --host 0.0.0.0 --port 8000` | `8004:8000` | postgres (healthy), redis (healthy), minio (healthy) |
+| `dashboard` | `frontend/Dockerfile` | nginx (built-in) | `3000:8080` | query-api (healthy) |
+
+### Port Summary
+
+| Port | Service | Protocol |
+|------|---------|----------|
+| 3000 | Dashboard (React UI) | HTTP |
+| 5432 | PostgreSQL | TCP |
+| 6379 | Redis | TCP |
+| 8001 | Symbol Registry API | HTTP |
+| 8002 | Trading Engine API | HTTP |
+| 8003 | Risk Engine API | HTTP |
+| 8004 | Query API | HTTP |
+| 8080 | Trino | HTTP |
+| 8088 | Superset | HTTP |
+| 9000 | MinIO API | HTTP |
+| 9001 | MinIO Console | HTTP |
+| 9083 | Hive Metastore | Thrift |
+| 11434 | Ollama | HTTP |
+
+---
+
+## Environment Variables
+
+### Shared Application Environment (`x-app-env`)
+
+All application services inherit these variables via the `x-app-env` YAML anchor:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `POSTGRES_HOST` | `postgres` | PostgreSQL hostname (Docker service name) |
+| `POSTGRES_PORT` | `5432` | PostgreSQL port |
+| `POSTGRES_DB` | `stonks` | Database name |
+| `POSTGRES_USER` | `stonks` | Database user |
+| `POSTGRES_PASSWORD` | `stonks_dev` | Database password |
+| `REDIS_HOST` | `redis` | Redis hostname (Docker service name) |
+| `REDIS_PORT` | `6379` | Redis port |
+| `MINIO_ENDPOINT` | `minio:9000` | MinIO API endpoint |
+| `MINIO_ACCESS_KEY` | `minioadmin` | MinIO access key |
+| `MINIO_SECRET_KEY` | `minioadmin` | MinIO secret key |
+| `OLLAMA_BASE_URL` | `http://ollama:11434` | Ollama LLM server URL |
+
+### `.env` File
+
+The `.env` file is loaded by `ingestion`, `broker-adapter`, and `trading-engine` via the `env_file` directive. Create it in the repository root:
+
+```dotenv
+# Stonks Oracle — Environment Variables
+# These are loaded by ingestion, broker-adapter, and trading-engine services.
+
+# Polygon.io market data API key (required for live data ingestion)
+MARKET_DATA_API_KEY=
+
+# Alpaca broker credentials (required for paper/live trading)
+BROKER_API_KEY=
+BROKER_API_SECRET=
+BROKER_BASE_URL=https://paper-api.alpaca.markets
+```
+
+| Variable | Required | Default | Used By | Description |
+|----------|----------|---------|---------|-------------|
+| `MARKET_DATA_API_KEY` | No* | (empty) | ingestion | Polygon.io API key for market data fetching |
+| `BROKER_API_KEY` | No* | (empty) | broker-adapter, trading-engine | Alpaca API key |
+| `BROKER_API_SECRET` | No* | (empty) | broker-adapter, trading-engine | Alpaca API secret |
+| `BROKER_BASE_URL` | No | `https://paper-api.alpaca.markets` | broker-adapter, trading-engine | Alpaca API base URL |
+
+*Services start without these keys but run in degraded mode — ingestion cannot fetch market data and the broker adapter cannot execute trades.
+
+### Infrastructure Service Environment
+
+**PostgreSQL** (`postgres`):
+
+| Variable | Value | Description |
+|----------|-------|-------------|
+| `POSTGRES_DB` | `stonks` | Database created on first start |
+| `POSTGRES_USER` | `stonks` | Superuser for the database |
+| `POSTGRES_PASSWORD` | `stonks_dev` | Password for the database user |
+
+**MinIO** (`minio`):
+
+| Variable | Value | Description |
+|----------|-------|-------------|
+| `MINIO_ROOT_USER` | `minioadmin` | MinIO admin username |
+| `MINIO_ROOT_PASSWORD` | `minioadmin` | MinIO admin password |
+
+**Trino** (`trino`):
+
+| Variable | Value | Description |
+|----------|-------|-------------|
+| `MINIO_ACCESS_KEY` | `minioadmin` | Passed to Trino for MinIO catalog access |
+| `MINIO_SECRET_KEY` | `minioadmin` | Passed to Trino for MinIO catalog access |
+
+**Hive Metastore** (`hive-metastore`):
+
+| Variable | Value | Description |
+|----------|-------|-------------|
+| `SERVICE_NAME` | `metastore` | Tells Hive to run in metastore-only mode |
+| `DB_DRIVER` | `derby` | Embedded Derby database for metadata |
+
+**Superset** (`superset`):
+
+| Variable | Value | Description |
+|----------|-------|-------------|
+| `SUPERSET_SECRET_KEY` | `stonks-dev-secret-key-change-me` | Flask secret key (change in production) |
+| `ADMIN_USERNAME` | `admin` | Initial admin username |
+| `ADMIN_PASSWORD` | `admin` | Initial admin password |
+| `ADMIN_EMAIL` | `admin@stonks.local` | Initial admin email |
+
+### Additional Configuration Variables
+
+All application services support additional environment variables loaded via `services/shared/config.py`. These can be added to individual service `environment` blocks or to the `x-app-env` anchor as needed:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `REDIS_DB` | `0` | Redis database number |
+| `REDIS_PASSWORD` | (none) | Redis password (not needed in Docker Compose) |
+| `MINIO_SECURE` | `false` | Use HTTPS for MinIO |
+| `OLLAMA_MODEL` | `qwen3.5:9b` | Default LLM model for extraction |
+| `OLLAMA_TIMEOUT` | `120` | Ollama request timeout (seconds) |
+| `OLLAMA_MAX_RETRIES` | `2` | Max retries for Ollama requests |
+| `TRINO_HOST` | `localhost` | Trino hostname |
+| `TRINO_PORT` | `8080` | Trino port |
+| `TRINO_CATALOG` | `lakehouse` | Trino catalog name |
+| `TRINO_SCHEMA` | `stonks` | Trino schema name |
+| `MARKET_DATA_BASE_URL` | `https://api.polygon.io` | Polygon.io base URL |
+| `MARKET_DATA_PROVIDER` | `polygon` | Market data provider |
+| `BROKER_MODE` | `paper` | Broker mode: `paper` or `live` |
+| `BROKER_PROVIDER` | `alpaca` | Broker provider |
+| `TRADING_ENABLED` | `false` | Enable autonomous trading engine |
+| `TRADING_RISK_TIER` | `moderate` | Risk tier: `conservative`, `moderate`, `aggressive` |
+| `TRADING_POLLING_INTERVAL_SECONDS` | `60` | Recommendation polling interval |
+| `TRADING_MAX_OPEN_POSITIONS` | `10` | Maximum concurrent open positions |
+| `MACRO_ENABLED` | `true` | Enable macro signal layer |
+| `COMPETITIVE_ENABLED` | `true` | Enable competitive signal layer |
+| `LOG_LEVEL` | `INFO` | Logging level |
+| `JSON_LOGS` | `true` | Enable structured JSON logging |
+| `DEPLOY_STAGE` | (empty) | Deployment stage prefix for bucket names |
+
+See `services/shared/config.py` for the complete list of all supported environment variables with their defaults.
+
+---
+
+## Volume Mounts and Data Persistence
+
+Docker Compose defines five named volumes for persistent data:
+
+| Volume | Mounted By | Mount Path | Contents |
+|--------|-----------|------------|----------|
+| `pgdata` | postgres | `/var/lib/postgresql/data` | PostgreSQL database files |
+| `miniodata` | minio | `/data` | MinIO object storage (raw artifacts, lakehouse Parquet files) |
+| `ollama_models` | ollama | `/root/.ollama` | Downloaded LLM model weights |
+| `hive_data` | hive-metastore | `/opt/hive/data` | Hive metastore Derby database |
+| `superset_data` | superset | `/app/superset_home` | Superset configuration and metadata |
+
+### Bind Mounts
+
+In addition to named volumes, several services use bind mounts for configuration:
+
+| Service | Host Path | Container Path | Mode | Purpose |
+|---------|-----------|---------------|------|---------|
+| postgres | `./infra/migrations` | `/docker-entrypoint-initdb.d` | rw | SQL migrations auto-applied on first start |
+| trino | `./infra/trino/catalog` | `/etc/trino/catalog` | rw | Trino catalog configuration (lakehouse, iceberg) |
+| hive-metastore | `./infra/hive/core-site.xml` | `/opt/hive/conf/core-site.xml` | ro | Hadoop core-site config for MinIO access |
+| hive-metastore | `./infra/hive/metastore-site.xml` | `/opt/hive/conf/metastore-site.xml` | ro | Hive metastore config |
+
+### Resetting Data
+
+To destroy all persistent data and start fresh:
+
+```bash
+# Stop all containers and remove named volumes
+docker compose down -v
+```
+
+This removes `pgdata`, `miniodata`, `ollama_models`, `hive_data`, and `superset_data`. The next `docker compose up` will re-initialize PostgreSQL with migrations, re-create MinIO buckets (via `minio-init`), and re-download Ollama models.
+
+To reset only specific volumes:
+
+```bash
+docker compose down
+docker volume rm stonks-oracle_pgdata # Reset database only
+docker compose up -d
+```
+
+> **Note**: Volume names are prefixed with the project directory name (e.g., `stonks-oracle_pgdata`). Use `docker volume ls` to see exact names.
+
+---
+
+## Health Checks
+
+Every service has a health check configured. Docker Compose uses these to enforce startup ordering via `depends_on` with `condition: service_healthy`.
+
+### Infrastructure Health Checks
+
+| Service | Test Command | Interval | Retries |
+|---------|-------------|----------|---------|
+| `postgres` | `pg_isready -U stonks` | 5s | 5 |
+| `redis` | `redis-cli ping` | 5s | 5 |
+| `minio` | `mc ready local` | 5s | 5 |
+
+### Application Health Checks — FastAPI Services
+
+FastAPI services (symbol-registry, trading-engine, risk-engine, query-api) use HTTP health endpoints:
+
+| Service | Test Command | Interval | Timeout | Retries | Start Period |
+|---------|-------------|----------|---------|---------|-------------|
+| `symbol-registry` | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
+| `trading-engine` | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
+| `risk-engine` | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
+| `query-api` | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
+| `dashboard` | `curl -f http://localhost:8080/` | 10s | 5s | 3 | 10s |
+
+### Application Health Checks — Worker Services
+
+Worker services (no HTTP endpoint) use process liveness checks:
+
+| Service | Test Command | Interval | Timeout | Retries | Start Period |
+|---------|-------------|----------|---------|---------|-------------|
+| `scheduler` | `pgrep -f 'python -m services.scheduler.app'` | 10s | 5s | 3 | 15s |
+| `ingestion` | `pgrep -f 'python -m services.ingestion.worker'` | 10s | 5s | 3 | 15s |
+| `parser` | `pgrep -f 'python -m services.parser.worker'` | 10s | 5s | 3 | 15s |
+| `extractor` | `pgrep -f 'python -m services.extractor.main'` | 10s | 5s | 3 | 15s |
+| `aggregation` | `pgrep -f 'python -m services.aggregation.main'` | 10s | 5s | 3 | 15s |
+| `recommendation` | `pgrep -f 'python -m services.recommendation.main'` | 10s | 5s | 3 | 15s |
+| `broker-adapter` | `pgrep -f 'python -m services.adapters.broker_service'` | 10s | 5s | 3 | 15s |
+| `lake-publisher` | `pgrep -f 'python -m services.lake_publisher.jobs'` | 10s | 5s | 3 | 15s |
+
+### Verifying Service Health
+
+```bash
+# Check all service statuses
+docker compose ps
+
+# Check a specific service
+docker compose ps query-api
+
+# Inspect health check details for a container
+docker inspect --format='{{json .State.Health}}' stonks-oracle-query-api-1 | python -m json.tool
+```
+
+---
+
+## Dockerfile Build Details
+
+### `docker/Dockerfile` — Generic Python Service Image
+
+Used by all application services except the scheduler. Accepts a `SERVICE_CMD` build argument that determines which service the container runs.
+
+**Base image**: `python:3.12-slim`
+
+**Build arguments**:
+
+| Argument | Default | Description |
+|----------|---------|-------------|
+| `SERVICE_CMD` | `python -m services.scheduler.app` | The command executed when the container starts |
+
+**What gets copied**:
+- `requirements.txt` → pip dependencies installed
+- `services/` → all service source code
+- `tests/` → test files (available for in-container testing)
+- `conftest.py` → pytest configuration
+
+**Environment variables set**:
+- `PYTHONDONTWRITEBYTECODE=1` — no `.pyc` files
+- `PYTHONUNBUFFERED=1` — unbuffered stdout/stderr for log visibility
+- `PYTHONPATH=/app` — ensures `services.*` imports resolve
+
+**System packages installed**: `gcc`, `libpq-dev` (PostgreSQL client library), `curl` (for health checks)
+
+**Security**: Runs as non-root user `stonks` (UID 1000).
+
+**How `SERVICE_CMD` works**: The `CMD` directive is `sh -c "${SERVICE_CMD}"`, so the build argument becomes the runtime command. Each service in `docker-compose.yml` overrides this via the `args.SERVICE_CMD` build parameter:
+
+```yaml
+query-api:
+ build:
+ context: .
+ dockerfile: docker/Dockerfile
+ args:
+ SERVICE_CMD: "uvicorn services.api.app:app --host 0.0.0.0 --port 8000"
+```
+
+### `docker/Dockerfile.scheduler` — Scheduler Image
+
+A specialized variant of the generic Dockerfile used only by the `scheduler` service. Adds `postgresql-client` for running database migrations via `psql`.
+
+**Additional contents**:
+- `infra/migrations/` → copied to `/app/infra/migrations/` for migration execution
+- `postgresql-client` system package installed
+
+**Command**: Hardcoded `CMD ["python", "-m", "services.scheduler.app"]` (no `SERVICE_CMD` argument).
+
+### `docker/Dockerfile.superset` — Custom Superset Image
+
+Extends the official Apache Superset image with additional database drivers.
+
+**Base image**: `apache/superset:latest`
+
+**Additional packages**: `trino[sqlalchemy]`, `psycopg2-binary`, `redis`
+
+### `frontend/Dockerfile` — Dashboard Image
+
+Multi-stage build for the React dashboard.
+
+**Stage 1 — Build** (base: `node:24-alpine`):
+
+| Build Argument | Default | Description |
+|---------------|---------|-------------|
+| `VITE_QUERY_API_URL` | `""` | Query API base URL (empty = use relative `/api/` proxy) |
+| `VITE_SYMBOL_REGISTRY_URL` | `""` | Symbol Registry base URL (empty = use relative `/registry/` proxy) |
+| `VITE_RISK_ENGINE_URL` | `""` | Risk Engine base URL (empty = use relative `/risk/` proxy) |
+
+**Stage 2 — Serve** (base: `nginxinc/nginx-unprivileged:alpine`):
+- Serves the built static files on port 8080
+- Uses `frontend/nginx.conf` for SPA fallback and API reverse proxying
+- Proxies `/api/` → `query-api:8000`, `/registry/` → `symbol-registry:8000`, `/risk/` → `risk-engine:8000`, `/trading/` → `trading-engine:8000`
+
+### Building Custom Images
+
+To build a single service image locally:
+
+```bash
+# Build the query-api image
+docker compose build query-api
+
+# Build with a custom SERVICE_CMD
+docker build -t my-custom-service \
+ --build-arg SERVICE_CMD="python -m services.my_service.main" \
+ -f docker/Dockerfile .
+
+# Build the dashboard with custom API URLs
+docker build -t my-dashboard \
+ --build-arg VITE_QUERY_API_URL="https://api.example.com" \
+ -f frontend/Dockerfile frontend/
+
+# Rebuild all images
+docker compose build
+```
+
+---
+
+## Dependency Ordering
+
+Docker Compose enforces startup order using `depends_on` with health check conditions. The dependency graph is:
+
+```
+postgres (healthy) ──┬── scheduler
+ ├── symbol-registry
+ ├── ingestion
+ ├── parser
+ ├── extractor
+ ├── aggregation
+ ├── recommendation
+ ├── trading-engine
+ ├── risk-engine
+ ├── broker-adapter
+ ├── lake-publisher
+ └── query-api
+
+redis (healthy) ─────┬── scheduler
+ ├── ingestion
+ ├── parser
+ ├── extractor
+ ├── aggregation
+ ├── recommendation
+ ├── trading-engine
+ ├── broker-adapter
+ └── query-api
+
+minio (healthy) ─────┬── minio-init
+ ├── ingestion
+ ├── lake-publisher
+ └── query-api
+
+ollama (started) ────── extractor
+
+minio ───────────────── trino
+hive-metastore ─────── trino
+trino ──────────────── superset (via depends_on)
+
+query-api (healthy) ── dashboard
+```
+
+Services with `condition: service_healthy` wait until the dependency's health check passes. The `extractor` depends on `ollama` with `condition: service_started` (no health check — Ollama may take time to load models).
+
+---
+
+## Operational Commands
+
+### Starting Services
+
+```bash
+# Start all services in the background
+docker compose up -d
+
+# Start only infrastructure (useful for local development)
+docker compose up -d postgres redis minio minio-init ollama
+
+# Start a specific service and its dependencies
+docker compose up -d query-api
+```
+
+### Stopping Services
+
+```bash
+# Stop all services (preserves volumes)
+docker compose down
+
+# Stop all services and remove volumes (full reset)
+docker compose down -v
+
+# Stop a specific service
+docker compose stop trading-engine
+```
+
+### Restarting Services
+
+```bash
+# Restart a specific service
+docker compose restart query-api
+
+# Restart with a fresh build
+docker compose up -d --build query-api
+
+# Force recreate a service (picks up compose file changes)
+docker compose up -d --force-recreate query-api
+```
+
+### Viewing Logs
+
+```bash
+# Follow logs for all services
+docker compose logs -f
+
+# Follow logs for a specific service
+docker compose logs -f query-api
+
+# View last 50 lines of a service's logs
+docker compose logs --tail=50 ingestion
+
+# View logs for multiple services
+docker compose logs -f scheduler ingestion extractor
+```
+
+### Scaling Replicas
+
+```bash
+# Scale a worker service to 3 replicas
+docker compose up -d --scale ingestion=3
+
+# Scale multiple services
+docker compose up -d --scale ingestion=3 --scale extractor=2
+
+# Scale back to 1
+docker compose up -d --scale ingestion=1
+```
+
+> **Note**: Scaling works best for worker services (ingestion, parser, extractor, aggregation, recommendation, broker-adapter, lake-publisher) that consume from Redis queues. Do not scale FastAPI services that expose host ports without adjusting port mappings.
+
+### Inspecting Services
+
+```bash
+# List all services and their status
+docker compose ps
+
+# View resource usage
+docker compose top
+
+# Execute a command inside a running container
+docker compose exec query-api python -c "from services.shared.config import load_config; print(load_config())"
+
+# Open a shell in a container
+docker compose exec postgres psql -U stonks -d stonks
+```
+
+### Full Reset
+
+```bash
+# Nuclear option: stop everything, remove volumes, rebuild, restart
+docker compose down -v
+docker compose build --no-cache
+docker compose up -d
+```
+
+This destroys all data (database, object storage, model weights, metastore, Superset config) and starts from scratch. PostgreSQL migrations are re-applied automatically. MinIO buckets are re-created by `minio-init`. Ollama models must be re-downloaded.
+
+---
+
+## MinIO Bucket Initialization
+
+The `minio-init` service runs once on startup and creates the required object storage buckets:
+
+| Bucket | Purpose |
+|--------|---------|
+| `stonks-raw-market` | Raw market data from Polygon.io |
+| `stonks-raw-news` | Raw news articles |
+| `stonks-raw-filings` | Raw SEC filings |
+| `stonks-normalized` | Normalized/parsed documents |
+| `stonks-llm-prompts` | LLM prompt archives |
+| `stonks-llm-results` | LLM extraction results |
+| `stonks-lakehouse` | Parquet fact tables for Trino |
+| `stonks-audit` | Audit trail artifacts |
+
+Access the MinIO console at `http://localhost:9001` (credentials: `minioadmin` / `minioadmin`).
+
+---
+
+## Dashboard Reverse Proxy
+
+The dashboard container runs nginx with reverse proxy rules that route API requests to backend services using Docker Compose service names:
+
+| Path | Proxied To | Service |
+|------|-----------|---------|
+| `/api/` | `http://query-api:8000` | Query API |
+| `/registry/` | `http://symbol-registry:8000/` | Symbol Registry API |
+| `/risk/` | `http://risk-engine:8000/` | Risk Engine API |
+| `/trading/` | `http://trading-engine:8000/` | Trading Engine API |
+
+All other paths serve the React SPA with `try_files` fallback to `index.html`.
+
+---
+
+## Troubleshooting
+
+### Service won't start
+
+Check dependency health:
+
+```bash
+docker compose ps postgres redis minio
+```
+
+If infrastructure services are unhealthy, application services will wait indefinitely. Check infrastructure logs:
+
+```bash
+docker compose logs postgres
+```
+
+### Database migration errors
+
+Migrations in `./infra/migrations/` are applied by PostgreSQL's `docker-entrypoint-initdb.d` mechanism, which only runs on first database initialization. If you need to re-run migrations:
+
+```bash
+docker compose down -v # Remove pgdata volume
+docker compose up -d # Migrations re-applied on fresh init
+```
+
+### Ollama model not available
+
+The extractor service needs an LLM model loaded in Ollama. Pull a model manually:
+
+```bash
+docker compose exec ollama ollama pull qwen3.5:9b
+```
+
+### Port conflicts
+
+If a port is already in use, modify the host port mapping in `docker-compose.yml`:
+
+```yaml
+query-api:
+ ports:
+ - "9004:8000" # Changed from 8004 to 9004
+```
diff --git a/docs/helm-reference.md b/docs/helm-reference.md
new file mode 100644
index 0000000..2a1edd6
--- /dev/null
+++ b/docs/helm-reference.md
@@ -0,0 +1,659 @@
+# Helm Chart Configuration Reference
+
+Complete reference for the Stonks Oracle Helm chart at `infra/helm/stonks-oracle/`.
+
+| | |
+|---|---|
+| **Chart name** | `stonks-oracle` |
+| **Chart version** | `0.1.0` |
+| **App version** | `1.0.0` |
+| **Chart type** | `application` |
+
+Install with:
+
+```bash
+helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle
+```
+
+Override values per stage:
+
+```bash
+# Beta
+helm upgrade --install stonks-oracle infra/helm/stonks-oracle \
+ -n stonks-oracle-beta -f infra/helm/stonks-oracle/values-beta.yaml
+
+# Paper trading
+helm upgrade --install stonks-oracle infra/helm/stonks-oracle \
+ -n stonks-oracle -f infra/helm/stonks-oracle/values-paper.yaml
+```
+
+---
+
+## Table of Contents
+
+- [image — Global Image Settings](#image--global-image-settings)
+- [pipelineEnabled — Pipeline Toggle](#pipelineenabled--pipeline-toggle)
+- [services — Service Deployments](#services--service-deployments)
+- [config — ConfigMap Environment Variables](#config--configmap-environment-variables)
+- [secrets — Kubernetes Secrets](#secrets--kubernetes-secrets)
+- [ingress — Ingress Configuration](#ingress--ingress-configuration)
+- [Analytics Stack — Trino, Hive Metastore, Superset](#analytics-stack--trino-hive-metastore-superset)
+- [networkPolicies — Network Policy Configuration](#networkpolicies--network-policy-configuration)
+- [Value Override Files](#value-override-files)
+
+---
+
+## `image` — Global Image Settings
+
+Controls the container image registry, pull policy, and tag for all service deployments. Each service image is resolved as `{registry}/{service.image}:{tag}`.
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `image.registry` | string | `registry.celestium.life/stonks-oracle` | Container registry prefix. Each service appends its `image` name to this. |
+| `image.pullPolicy` | string | `Always` | Kubernetes `imagePullPolicy`. Use `Always` for latest-tag workflows. |
+| `image.tag` | string | `latest` | Image tag applied to all services. CI overrides this with the Git SHA via `--set image.tag=`. |
+
+Example override:
+
+```bash
+helm upgrade --install stonks-oracle infra/helm/stonks-oracle \
+ --set image.tag=abc1234
+```
+
+---
+
+## `pipelineEnabled` — Pipeline Toggle
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `pipelineEnabled` | bool | `true` | Master toggle for the data pipeline. |
+
+When `false`, all services with `pipeline: true` in their definition are scaled to **0 replicas**. API-tier and trading-tier services continue running normally.
+
+**Affected services** (scaled to 0 when disabled): scheduler, ingestion, parser, extractor, aggregation, recommendation, broker-adapter, lake-publisher.
+
+**Unaffected services** (always run): symbol-registry, query-api, trading-engine, risk-engine, dashboard.
+
+The replica count logic in the deployment template:
+
+```yaml
+replicas: {{ if and (hasKey $svc "pipeline") $svc.pipeline (not .Values.pipelineEnabled) }}0{{ else }}{{ $svc.replicas }}{{ end }}
+```
+
+---
+
+## `services` — Service Deployments
+
+Each key under `services` defines a Kubernetes Deployment. The deployments template iterates over all entries and creates a Deployment + optional Service for each.
+
+### Per-Service Structure
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `replicas` | int | yes | Number of pod replicas. Set to 0 by `pipelineEnabled: false` for pipeline services. |
+| `image` | string | yes | Image name appended to `image.registry`. Also used as the Deployment name and pod label (`app: `). |
+| `command` | string | no | Shell command passed as `["sh", "-c", ""]`. Omit for images with a built-in entrypoint (e.g., dashboard/nginx). |
+| `tier` | string | yes | Service tier label (`stonks-oracle/tier`). One of: `api`, `frontend`, `processing`, `trading`, `orchestration`, `analytics`, `ingestion`. |
+| `port` | int | no | Container port. When set, a Kubernetes Service is created mapping `port → port`. |
+| `pipeline` | bool | no | If `true`, replicas are set to 0 when `pipelineEnabled` is `false`. |
+| `secrets` | list(string) | no | List of Secret names to mount via `envFrom.secretRef`. |
+| `resources` | object | yes | Kubernetes resource requests and limits (`cpu`, `memory`). |
+| `probes.readiness` | object | no | HTTP readiness probe: `path`, `port`, `initialDelay`, `period`. |
+| `probes.liveness` | object | no | HTTP liveness probe: `path`, `port`, `initialDelay`, `period`. |
+
+### Service Definitions
+
+#### scheduler
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `pipeline` | `true` |
+| `image` | `scheduler` |
+| `command` | `python -m services.scheduler.app` |
+| `tier` | `orchestration` |
+| `port` | — |
+| `secrets` | `stonks-core-secrets` |
+| `resources.requests` | cpu: 50m, memory: 64Mi |
+| `resources.limits` | cpu: 200m, memory: 128Mi |
+| `probes` | — |
+
+The scheduler deployment has two init containers (not configurable via values):
+1. **run-migrations** — applies all SQL files from `infra/migrations/*.sql` in sorted order.
+2. **seed-if-empty** — runs `python -m services.symbol_registry.seed` if the `companies` table is empty.
+
+#### symbolRegistry
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `image` | `symbol-registry` |
+| `command` | `uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000` |
+| `tier` | `api` |
+| `port` | `8000` |
+| `secrets` | `stonks-core-secrets` |
+| `resources.requests` | cpu: 100m, memory: 128Mi |
+| `resources.limits` | cpu: 500m, memory: 256Mi |
+| `probes.readiness` | path: `/docs`, port: 8000, initialDelay: 5s, period: 10s |
+| `probes.liveness` | path: `/docs`, port: 8000, initialDelay: 10s, period: 30s |
+
+#### ingestion
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `2` |
+| `pipeline` | `true` |
+| `image` | `ingestion` |
+| `command` | `python -m services.ingestion.worker` |
+| `tier` | `ingestion` |
+| `port` | — |
+| `secrets` | `stonks-core-secrets`, `stonks-market-secrets`, `stonks-broker-secrets` |
+| `resources.requests` | cpu: 100m, memory: 128Mi |
+| `resources.limits` | cpu: 500m, memory: 256Mi |
+
+#### parser
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `2` |
+| `pipeline` | `true` |
+| `image` | `parser` |
+| `command` | `python -m services.parser.worker` |
+| `tier` | `processing` |
+| `port` | — |
+| `secrets` | `stonks-core-secrets` |
+| `resources.requests` | cpu: 100m, memory: 128Mi |
+| `resources.limits` | cpu: 500m, memory: 256Mi |
+
+#### extractor
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `pipeline` | `true` |
+| `image` | `extractor` |
+| `command` | `python -m services.extractor.main` |
+| `tier` | `processing` |
+| `port` | — |
+| `secrets` | `stonks-core-secrets` |
+| `resources.requests` | cpu: 200m, memory: 256Mi |
+| `resources.limits` | cpu: 1, memory: 512Mi |
+
+Single replica is recommended — the extractor is bottlenecked by the shared Ollama GPU.
+
+#### aggregation
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `4` |
+| `pipeline` | `true` |
+| `image` | `aggregation` |
+| `command` | `python -m services.aggregation.main` |
+| `tier` | `processing` |
+| `port` | — |
+| `secrets` | `stonks-core-secrets` |
+| `resources.requests` | cpu: 100m, memory: 128Mi |
+| `resources.limits` | cpu: 500m, memory: 256Mi |
+
+#### recommendation
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `pipeline` | `true` |
+| `image` | `recommendation` |
+| `command` | `python -m services.recommendation.main` |
+| `tier` | `processing` |
+| `port` | — |
+| `secrets` | `stonks-core-secrets` |
+| `resources.requests` | cpu: 100m, memory: 128Mi |
+| `resources.limits` | cpu: 500m, memory: 256Mi |
+
+#### tradingEngine
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `image` | `trading-engine` |
+| `command` | `uvicorn services.trading.app:app --host 0.0.0.0 --port 8000` |
+| `tier` | `trading` |
+| `port` | `8000` |
+| `secrets` | `stonks-core-secrets`, `stonks-broker-secrets`, `stonks-gmail-secrets` |
+| `resources.requests` | cpu: 100m, memory: 256Mi |
+| `resources.limits` | cpu: 500m, memory: 512Mi |
+| `probes.readiness` | path: `/ready`, port: 8000, initialDelay: 5s, period: 10s |
+| `probes.liveness` | path: `/health`, port: 8000, initialDelay: 10s, period: 30s |
+
+#### riskEngine
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `image` | `risk` |
+| `command` | `uvicorn services.risk.app:app --host 0.0.0.0 --port 8000` |
+| `tier` | `trading` |
+| `port` | `8000` |
+| `secrets` | `stonks-core-secrets`, `stonks-broker-secrets` |
+| `resources.requests` | cpu: 100m, memory: 128Mi |
+| `resources.limits` | cpu: 500m, memory: 256Mi |
+
+#### brokerAdapter
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `pipeline` | `true` |
+| `image` | `broker-adapter` |
+| `command` | `python -m services.adapters.broker_service` |
+| `tier` | `trading` |
+| `port` | — |
+| `secrets` | `stonks-core-secrets`, `stonks-broker-secrets` |
+| `resources.requests` | cpu: 50m, memory: 64Mi |
+| `resources.limits` | cpu: 200m, memory: 128Mi |
+
+#### lakePublisher
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `pipeline` | `true` |
+| `image` | `lake-publisher` |
+| `command` | `python -m services.lake_publisher.jobs` |
+| `tier` | `analytics` |
+| `port` | — |
+| `secrets` | `stonks-core-secrets` |
+| `resources.requests` | cpu: 100m, memory: 128Mi |
+| `resources.limits` | cpu: 500m, memory: 256Mi |
+
+#### queryApi
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `image` | `query-api` |
+| `command` | `uvicorn services.api.app:app --host 0.0.0.0 --port 8000` |
+| `tier` | `api` |
+| `port` | `8000` |
+| `secrets` | `stonks-core-secrets` |
+| `resources.requests` | cpu: 100m, memory: 128Mi |
+| `resources.limits` | cpu: 500m, memory: 256Mi |
+| `probes.readiness` | path: `/docs`, port: 8000, initialDelay: 5s, period: 10s |
+
+#### dashboard
+
+| Field | Value |
+|-------|-------|
+| `replicas` | `1` |
+| `image` | `dashboard` |
+| `command` | — (nginx built-in entrypoint) |
+| `tier` | `frontend` |
+| `port` | `8080` |
+| `secrets` | — |
+| `resources.requests` | cpu: 50m, memory: 64Mi |
+| `resources.limits` | cpu: 200m, memory: 128Mi |
+| `probes.readiness` | path: `/`, port: 8080, initialDelay: 3s, period: 10s |
+| `probes.liveness` | path: `/`, port: 8080, initialDelay: 5s, period: 30s |
+
+---
+
+## `config` — ConfigMap Environment Variables
+
+All keys under `config` are rendered into a Kubernetes ConfigMap named `stonks-config` and injected into every service pod via `envFrom.configMapRef`. Values are strings.
+
+### Database
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `config.POSTGRES_HOST` | string | `postgresql-rw.postgresql-service.svc.cluster.local` | PostgreSQL hostname. Points to the CloudNativePG read-write service. |
+| `config.POSTGRES_PORT` | string | `5432` | PostgreSQL port. |
+| `config.POSTGRES_DB` | string | `stonks` | Database name. Override per stage (e.g., `stonks_beta`, `stonks_paper`). |
+| `config.POSTGRES_USER` | string | `stonks` | Database user. Override per stage. |
+| `config.REDIS_HOST` | string | `redis-master.redis-service.svc.cluster.local` | Redis hostname. |
+| `config.REDIS_PORT` | string | `6379` | Redis port. |
+| `config.REDIS_DB` | string | `0` | Redis database index. Use different indices per stage to isolate keys (beta: `1`, paper: `2`). |
+
+### Object Storage
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `config.MINIO_ENDPOINT` | string | `minio.minio-service.svc.cluster.local:80` | MinIO API endpoint (host:port). |
+| `config.MINIO_SECURE` | string | `false` | Use HTTPS for MinIO connections. Set to `true` if MinIO has TLS. |
+
+### LLM / Ollama
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `config.OLLAMA_BASE_URL` | string | `""` (empty) | Ollama API base URL. Set to the cluster-internal or external Ollama endpoint. |
+| `config.OLLAMA_MODEL` | string | `qwen3.5:9b-fast` | Default LLM model for extraction and classification agents. |
+| `config.OLLAMA_TIMEOUT` | string | `240` | Request timeout in seconds for Ollama API calls. |
+| `config.OLLAMA_MAX_RETRIES` | string | `2` | Maximum retry attempts for failed Ollama requests. |
+| `config.OLLAMA_RETRY_BASE_DELAY` | string | `1.0` | Base delay in seconds for exponential backoff on Ollama retries. |
+| `config.OLLAMA_RETRY_MAX_DELAY` | string | `10.0` | Maximum delay cap in seconds for Ollama retry backoff. |
+| `config.OLLAMA_RETRY_BACKOFF_MULTIPLIER` | string | `2.0` | Multiplier for exponential backoff between Ollama retries. |
+
+### Analytics / Trino
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `config.TRINO_HOST` | string | `trino.stonks-oracle.svc.cluster.local` | Trino coordinator hostname. |
+| `config.TRINO_PORT` | string | `8080` | Trino coordinator port. |
+| `config.TRINO_CATALOG` | string | `lakehouse` | Default Trino catalog for Hive-based queries. |
+| `config.TRINO_SCHEMA` | string | `stonks` | Default Trino schema. |
+| `config.TRINO_ICEBERG_CATALOG` | string | `iceberg` | Trino catalog for Iceberg table queries. |
+
+### Broker / Trading
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `config.BROKER_MODE` | string | `paper` | Broker execution mode. `paper` for simulated trading, `live` for real orders. |
+| `config.BROKER_PROVIDER` | string | `""` (empty) | Broker provider name (e.g., `alpaca`). |
+| `config.MARKET_DATA_BASE_URL` | string | `""` (empty) | Market data API base URL (e.g., `https://api.polygon.io`). |
+| `config.MARKET_DATA_PROVIDER` | string | `polygon` | Market data provider identifier. |
+| `config.TRADING_ENABLED` | string | `true` | Master toggle for the trading engine. Set to `false` to disable order submission. |
+| `config.TRADING_RISK_TIER` | string | `moderate` | Default risk tier for position sizing. Options: `conservative`, `moderate`, `aggressive`. |
+| `config.TRADING_ABSOLUTE_POSITION_CAP` | string | `10000.0` | Maximum dollar value per position. |
+| `config.TRADING_MAX_OPEN_POSITIONS` | string | `10` | Maximum number of concurrent open positions. |
+
+### Data Retention
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `config.RETENTION_RAW_MARKET_DAYS` | string | `90` | Days to retain raw market data before cleanup. |
+| `config.RETENTION_RAW_NEWS_DAYS` | string | `180` | Days to retain raw news articles. |
+| `config.RETENTION_RAW_FILINGS_DAYS` | string | `365` | Days to retain raw SEC filings. |
+| `config.RETENTION_NORMALIZED_DAYS` | string | `180` | Days to retain normalized/parsed documents. |
+| `config.RETENTION_LLM_PROMPTS_DAYS` | string | `365` | Days to retain LLM prompt logs. |
+| `config.RETENTION_LLM_RESULTS_DAYS` | string | `365` | Days to retain LLM extraction results. |
+| `config.RETENTION_LAKEHOUSE_DAYS` | string | `730` | Days to retain lakehouse fact tables. |
+| `config.RETENTION_AUDIT_DAYS` | string | `730` | Days to retain audit trail events. |
+| `config.RETENTION_CLEANUP_INTERVAL_HOURS` | string | `24` | Hours between retention cleanup runs. |
+| `config.RETENTION_BATCH_SIZE` | string | `1000` | Number of rows deleted per cleanup batch. |
+
+### Logging and Deployment
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `config.LOG_LEVEL` | string | `INFO` | Python logging level. Options: `DEBUG`, `INFO`, `WARNING`, `ERROR`. |
+| `config.JSON_LOGS` | string | `true` | Emit structured JSON logs when `true`. |
+| `config.DEPLOY_STAGE` | string | `""` (empty) | Deployment stage identifier. Used to isolate Redis keys and MinIO buckets per stage (e.g., `beta`, `paper`). |
+
+### Alerting
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `config.ALERT_SOURCE_FAILURE_THRESHOLD` | string | `3` | Number of consecutive source failures before firing an alert. |
+| `config.ALERT_SOURCE_FAILURE_WINDOW_HOURS` | string | `6` | Time window (hours) for evaluating source failure count. |
+| `config.ALERT_SCHEMA_FAILURE_RATE_THRESHOLD` | string | `0.3` | Schema validation failure rate (0.0–1.0) that triggers an alert. |
+| `config.ALERT_SCHEMA_FAILURE_WINDOW_HOURS` | string | `1` | Time window (hours) for evaluating schema failure rate. |
+| `config.ALERT_LAKE_LAG_THRESHOLD_MINUTES` | string | `60` | Minutes of lakehouse publish lag before alerting. |
+| `config.ALERT_BROKER_ERROR_THRESHOLD` | string | `3` | Number of broker errors before firing an alert. |
+| `config.ALERT_BROKER_ERROR_WINDOW_HOURS` | string | `1` | Time window (hours) for evaluating broker error count. |
+| `config.ALERT_CHECK_INTERVAL_SECONDS` | string | `120` | Seconds between alert evaluation cycles. |
+
+---
+
+## `secrets` — Kubernetes Secrets
+
+Secrets are rendered into five Kubernetes Secret objects. In the base `values.yaml`, all secret values default to empty strings. Inject real values at deploy time using `--set` flags or a values override file.
+
+### Secret Objects
+
+| Secret Name | Values Key | Consumed By |
+|-------------|-----------|-------------|
+| `stonks-core-secrets` | `secrets.core` | All services |
+| `stonks-broker-secrets` | `secrets.broker` | ingestion, trading-engine, risk-engine, broker-adapter |
+| `stonks-market-secrets` | `secrets.market` | ingestion |
+| `stonks-gmail-secrets` | `secrets.gmail` | trading-engine |
+| `stonks-dashboard-secrets` | `secrets.dashboard` | superset |
+
+### `secrets.core`
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `POSTGRES_PASSWORD` | string | `""` | PostgreSQL password. |
+| `MINIO_ACCESS_KEY` | string | `""` | MinIO access key (AWS-style). |
+| `MINIO_SECRET_KEY` | string | `""` | MinIO secret key. |
+| `REDIS_PASSWORD` | string | `""` | Redis authentication password. |
+
+### `secrets.broker`
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `BROKER_API_KEY` | string | `""` | Broker API key (e.g., Alpaca paper trading key). |
+| `BROKER_API_SECRET` | string | `""` | Broker API secret. |
+| `BROKER_BASE_URL` | string | `""` | Broker API base URL (e.g., `https://paper-api.alpaca.markets`). |
+
+### `secrets.market`
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `MARKET_DATA_API_KEY` | string | `""` | Market data provider API key (e.g., Polygon.io). |
+
+### `secrets.gmail`
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `GMAIL_SENDER` | string | `celes@celestium.life` | Gmail sender address for trading notifications. |
+| `GMAIL_RECIPIENT` | string | `celes@celestium.life` | Gmail recipient address for trading notifications. |
+| `GMAIL_APP_PASSWORD` | string | `""` | Gmail app password for SMTP authentication. |
+
+### `secrets.dashboard`
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `SUPERSET_SECRET_KEY` | string | `""` | Flask secret key for Superset session encryption. |
+| `SUPERSET_ADMIN_PASSWORD` | string | `""` | Superset admin user password. |
+
+### Injecting Secrets at Deploy Time
+
+```bash
+helm upgrade --install stonks-oracle infra/helm/stonks-oracle \
+ -n stonks-oracle \
+ --set secrets.core.POSTGRES_PASSWORD="" \
+ --set secrets.core.MINIO_ACCESS_KEY="" \
+ --set secrets.core.MINIO_SECRET_KEY="" \
+ --set secrets.core.REDIS_PASSWORD="" \
+ --set secrets.broker.BROKER_API_KEY="" \
+ --set secrets.broker.BROKER_API_SECRET="" \
+ --set secrets.broker.BROKER_BASE_URL="https://paper-api.alpaca.markets" \
+ --set secrets.market.MARKET_DATA_API_KEY="" \
+ --set secrets.gmail.GMAIL_APP_PASSWORD="" \
+ --set secrets.dashboard.SUPERSET_SECRET_KEY="" \
+ --set secrets.dashboard.SUPERSET_ADMIN_PASSWORD=""
+```
+
+---
+
+## `ingress` — Ingress Configuration
+
+Controls Traefik Ingress resources with TLS via cert-manager.
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `ingress.enabled` | bool | `true` | Create Ingress resources. Set to `false` for port-forward-only access. |
+| `ingress.className` | string | `traefik` | Kubernetes IngressClass name. |
+| `ingress.clusterIssuer` | string | `ca-issuer` | cert-manager ClusterIssuer for TLS certificates. |
+
+### Host Mappings
+
+| Key | Default | Routes To | Port |
+|-----|---------|-----------|------|
+| `ingress.hosts.queryApi` | `stonks-api.celestium.life` | query-api Service | 8000 |
+| `ingress.hosts.symbolRegistry` | `stonks-registry.celestium.life` | symbol-registry Service | 8000 |
+| `ingress.hosts.dashboard` | `stonks.celestium.life` | dashboard Service | 8080 |
+| `ingress.hosts.superset` | `stonks-dash.celestium.life` | superset Service | 8088 |
+| `ingress.hosts.trino` | `stonks-trino.celestium.life` | trino Service | 8080 |
+| `ingress.hosts.tradingEngine` | `stonks-trading.celestium.life` | trading-engine Service | 8000 |
+
+Setting `superset` or `trino` host to an empty string (`""`) disables that Ingress resource (the template uses a conditional check).
+
+Each Ingress resource gets a dedicated TLS secret (e.g., `stonks-api-tls`, `stonks-registry-tls`) automatically provisioned by cert-manager.
+
+---
+
+## Analytics Stack — Trino, Hive Metastore, Superset
+
+The analytics stack provides SQL-based querying over the lakehouse data stored in MinIO. Each component can be independently enabled or disabled.
+
+### `trino`
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `trino.enabled` | bool | `true` | Deploy the Trino coordinator. |
+| `trino.resources.requests.cpu` | string | `500m` | CPU request. |
+| `trino.resources.requests.memory` | string | `1Gi` | Memory request. |
+| `trino.resources.limits.cpu` | string | `2` | CPU limit. |
+| `trino.resources.limits.memory` | string | `4Gi` | Memory limit. |
+
+When enabled, Trino deploys with two auto-configured catalogs:
+- **`lakehouse`** — Hive connector for Parquet fact tables in MinIO.
+- **`iceberg`** — Iceberg connector for Iceberg-format tables.
+
+Both catalogs connect to the Hive Metastore for schema metadata and to MinIO for data via S3A. MinIO credentials are read from `stonks-core-secrets`.
+
+### `hiveMetastore`
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `hiveMetastore.enabled` | bool | `true` | Deploy the Hive Metastore. |
+| `hiveMetastore.storageSize` | string | `1Gi` | PersistentVolumeClaim size for the embedded Derby metastore database. |
+| `hiveMetastore.resources.requests.cpu` | string | `200m` | CPU request. |
+| `hiveMetastore.resources.requests.memory` | string | `512Mi` | Memory request. |
+| `hiveMetastore.resources.limits.cpu` | string | `1` | CPU limit. |
+| `hiveMetastore.resources.limits.memory` | string | `1Gi` | Memory limit. |
+
+Uses `apache/hive:4.0.0` with an embedded Derby database. The Thrift metastore listens on port 9083. MinIO credentials are injected from `stonks-core-secrets` via an init container that generates `core-site.xml` and `metastore-site.xml`.
+
+### `superset`
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `superset.enabled` | bool | `true` | Deploy Apache Superset. |
+| `superset.storageSize` | string | `2Gi` | PersistentVolumeClaim size for Superset home directory. |
+| `superset.resources.requests.cpu` | string | `200m` | CPU request. |
+| `superset.resources.requests.memory` | string | `512Mi` | Memory request. |
+| `superset.resources.limits.cpu` | string | `1` | CPU limit. |
+| `superset.resources.limits.memory` | string | `2Gi` | Memory limit. |
+
+Uses a custom image (`registry.celestium.life/stonks-oracle/superset`) with Trino and psycopg2 drivers pre-installed. Superset's metadata database is PostgreSQL (same cluster instance). Redis is used for caching. Credentials come from `stonks-core-secrets` and `stonks-dashboard-secrets`.
+
+Superset listens on port 8088 with a readiness probe at `/health`.
+
+### Disabling the Analytics Stack
+
+To disable the entire analytics stack (e.g., in beta environments):
+
+```yaml
+trino:
+ enabled: false
+hiveMetastore:
+ enabled: false
+superset:
+ enabled: false
+```
+
+---
+
+## `networkPolicies` — Network Policy Configuration
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `networkPolicies.enabled` | bool | `true` | Deploy NetworkPolicy resources. |
+
+When enabled, the chart creates a **default-deny-ingress** policy that blocks all inbound traffic to every pod in the namespace. Individual allow policies are then created for services that need ingress:
+
+| Policy | Target Pod | Allowed Sources | Port |
+|--------|-----------|-----------------|------|
+| `allow-query-api-ingress` | `query-api` | kube-system (Traefik), dashboard | 8000 |
+| `allow-symbol-registry-ingress` | `symbol-registry` | kube-system (Traefik), dashboard | 8000 |
+| `allow-risk-engine-ingress` | `risk` | broker-adapter, query-api, dashboard | 8000 |
+| `allow-trading-engine-ingress` | `trading-engine` | query-api, dashboard, kube-system (Traefik) | 8000 |
+| `allow-superset-ingress` | `superset` | kube-system (Traefik) | 8088 |
+| `allow-trino-ingress` | `trino` | superset, query-api, kube-system (Traefik) | 8080 |
+| `allow-hive-metastore-ingress` | `hive-metastore` | trino, lake-publisher | 9083 |
+| `allow-dashboard-ingress` | `dashboard` | kube-system (Traefik) | 8080 |
+| `deny-broker-adapter-ingress` | `broker-adapter` | (none — explicit deny) | — |
+
+The trading-engine also has egress rules allowing outbound connections to PostgreSQL (5432), Redis (6379), HTTPS (443), SMTP (587), and DNS (53).
+
+Pipeline workers (scheduler, ingestion, parser, extractor, aggregation, recommendation, lake-publisher) have no explicit ingress allow policies — they rely on the default-deny and communicate only via outbound connections to Redis queues and PostgreSQL.
+
+---
+
+## Value Override Files
+
+The chart ships with two override files for staged deployments. ArgoCD or Kargo applies these during promotion.
+
+### `values-beta.yaml` — Beta / Integration Testing
+
+**Purpose**: Integration testing environment deployed to `stonks-oracle-beta` namespace. Shares infrastructure with paper but uses isolated database (`stonks_beta`), Redis DB index (`1`), and separate ingress hostnames.
+
+Key overrides:
+
+| Key | Beta Value | Reason |
+|-----|-----------|--------|
+| `pipelineEnabled` | `true` | Services deployed (ArgoCD health checks), but pipeline defaults to OFF via `PIPELINE_DEFAULT_OFF`. |
+| `config.DEPLOY_STAGE` | `beta` | Isolates Redis keys (`stonks:beta:*`) and MinIO buckets (`beta-stonks-*`). |
+| `config.POSTGRES_DB` | `stonks_beta` | Separate database for beta data. |
+| `config.REDIS_DB` | `1` | Separate Redis DB index. |
+| `config.LOG_LEVEL` | `DEBUG` | Verbose logging for debugging. |
+| `config.TRADING_ENABLED` | `false` | Safety net — no order submission in beta. |
+| `config.PIPELINE_DEFAULT_OFF` | `true` | Scheduler won't enqueue jobs unless explicitly enabled. |
+| `config.OLLAMA_MODEL` | `qwen3.6` | May use a different model version for testing. |
+| `trino.enabled` | `false` | Analytics stack disabled in beta. |
+| `hiveMetastore.enabled` | `false` | Analytics stack disabled in beta. |
+| `superset.enabled` | `false` | Analytics stack disabled in beta. |
+
+Beta ingress hostnames:
+
+| Service | Hostname |
+|---------|----------|
+| Query API | `stonks-api-beta.celestium.life` |
+| Symbol Registry | `stonks-registry-beta.celestium.life` |
+| Dashboard | `stonks-beta.celestium.life` |
+| Trading Engine | `stonks-trading-beta.celestium.life` |
+| Superset | (disabled) |
+| Trino | (disabled) |
+
+### `values-paper.yaml` — Paper Trading
+
+**Purpose**: Paper trading environment with real market data but simulated order execution via Alpaca's paper trading API. Deployed to the main `stonks-oracle` namespace.
+
+Key overrides:
+
+| Key | Paper Value | Reason |
+|-----|-----------|--------|
+| `config.BROKER_MODE` | `paper` | Simulated order execution. |
+| `config.BROKER_PROVIDER` | `alpaca` | Alpaca paper trading API. |
+| `config.TRADING_ENABLED` | `true` | Trading engine active. |
+| `config.POSTGRES_DB` | `stonks_paper` | Separate database for paper trading data. |
+| `config.POSTGRES_USER` | `stonks_paper` | Separate database user. |
+| `config.REDIS_DB` | `2` | Separate Redis DB index. |
+| `config.DEPLOY_STAGE` | `paper` | Stage identifier. |
+| `config.LOG_LEVEL` | `INFO` | Standard logging. |
+| `services.extractor.replicas` | `1` | Single replica (GPU bottleneck). |
+
+Paper ingress hostnames:
+
+| Service | Hostname |
+|---------|----------|
+| Query API | `stonks-paper-api.celestium.life` |
+| Symbol Registry | `stonks-paper-registry.celestium.life` |
+| Dashboard | `stonks-paper.celestium.life` |
+| Superset | `stonks-paper-dash.celestium.life` |
+| Trino | `stonks-paper-trino.celestium.life` |
+| Trading Engine | `stonks-paper-trading.celestium.life` |
+
+### Deployment Stage Progression
+
+```
+values-beta.yaml values-paper.yaml values.yaml (base)
+ Beta → Paper Trading → Production
+ Integration Simulated orders Live trading
+ testing Real market data Real orders
+ Pipeline OFF Pipeline ON Pipeline ON
+ Trading OFF Trading ON Trading ON
+ Analytics OFF Analytics ON Analytics ON
+```
+
+Promotion between stages is managed by Kargo/ArgoCD. CI sets the image tag, and the promotion pipeline applies the appropriate values file.
diff --git a/docs/intelligence-pipeline-deep-dive/01-data-ingestion-and-preparation.md b/docs/intelligence-pipeline-deep-dive/01-data-ingestion-and-preparation.md
new file mode 100644
index 0000000..04ccaa3
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/01-data-ingestion-and-preparation.md
@@ -0,0 +1,130 @@
+# Page 1 — Data Ingestion and Preparation
+
+Every signal that Stonks Oracle eventually acts on begins its life as raw data pulled from an external source. Before any AI agent can extract structured intelligence, before any trend can accumulate, and before any trade can be placed, the platform must first discover new content, fetch it reliably, eliminate duplicates, store the raw artifacts for audit, and normalize the text into a form suitable for downstream processing. This page traces that journey from external API to parser output, covering the Scheduler, Ingestion Worker, deduplication layer, raw storage, and Parser in detail.
+
+For a visual overview of the full flow described here, see the [Ingestion to Extraction Flow diagram](diagrams/ingestion-to-extraction-flow.md).
+
+---
+
+## Four Categories of Input Data
+
+Stonks Oracle tracks 50 companies across 10 sectors, and it draws intelligence from four distinct categories of external data. Each category has its own adapter, its own API conventions, and its own scheduling cadence, but all of them feed into the same ingestion pipeline.
+
+The first category is **company news**, sourced from the Polygon.io ticker news endpoint (`/v2/reference/news`). The `PolygonNewsAdapter` in `services/adapters/news_adapter.py` fetches articles linked to a specific ticker, returning structured results that include title, publisher, article URL, description, keywords, and publication timestamp. Each request can return up to 1,000 articles, though the default limit is 20 per fetch. The adapter tracks the most recent `published_utc` value and uses it on subsequent fetches to avoid re-retrieving articles the system has already seen.
+
+The second category is **SEC filings**, sourced from the SEC EDGAR full-text search system (EFTS). The `SECEdgarAdapter` in `services/adapters/filings_adapter.py` queries the `/LATEST/search-index` endpoint for 8-K, 10-Q, 10-K, and other form types associated with a company's ticker or CIK number. Unlike the Polygon endpoints, EDGAR is a public API that requires no key — only a descriptive `User-Agent` header per the SEC's fair-access policy. The adapter deduplicates results by accession number (`adsh`), filters out non-primary documents like XML fragments and graphics, and constructs the SEC EDGAR filing index URL for each hit so downstream services can fetch the full document.
+
+The third category is **market data**, also sourced from Polygon.io. The `PolygonMarketAdapter` in `services/adapters/market_adapter.py` supports multiple endpoints: previous-day aggregate bars (`/v2/aggs/ticker/{ticker}/prev`), range bars for custom date windows, intraday hourly bars, grouped daily bars that return data for all tickers in a single call (`/v2/aggs/grouped/locale/us/market/stocks/{date}`), and ticker detail lookups. Market data follows a different path than textual content — it does not pass through the Parser or Extractor, since the structured numeric data is already in a usable form.
+
+The fourth category is **macro and geopolitical news**, fetched by the `MacroNewsAdapter` in `services/adapters/macro_news_adapter.py`. Unlike the other three categories, macro news is not company-specific. These sources have `source_type='macro_news'` in the `sources` database table and may have a `NULL` `company_id`. The adapter fetches from a configurable HTTP endpoint (typically the Polygon news API filtered for broad market topics) and returns articles that describe global events — trade policy shifts, central bank decisions, geopolitical conflicts — rather than company-specific developments. Macro news articles are eventually classified by the Global Event Classifier agent and routed through a separate queue, as described in [Page 2](02-ai-agent-processing-and-extraction.md).
+
+All four adapter classes inherit from `BaseAdapter` defined in `services/adapters/base.py` and return an `AdapterResult` dataclass containing the raw payload bytes, a SHA-256 content hash, a list of parsed item dicts, HTTP metadata (status code, response time), and an error field that is `None` on success. This uniform interface allows the Ingestion Worker to handle all source types through a single dispatch mechanism.
+
+---
+
+## The Scheduler: Orchestrating Ingestion Cycles
+
+The Scheduler (`services/scheduler/app.py`) is the heartbeat of the ingestion pipeline. It runs a continuous loop that ticks every 15 seconds (`SCHEDULER_TICK = 15`), and on each tick it evaluates which sources are due for their next fetch. The Scheduler does not fetch data itself — it enqueues jobs onto the `stonks:queue:ingestion` Redis list for the Ingestion Worker to process.
+
+Each source type has a default polling cadence defined in the `DEFAULT_CADENCES` dictionary:
+
+| Source Type | Default Cadence |
+|---------------|-----------------|
+| `market_api` | 300 seconds |
+| `news_api` | 300 seconds |
+| `filings_api` | 3,600 seconds |
+| `macro_news` | 600 seconds |
+| `web_scrape` | 1,800 seconds |
+| `broker` | 30 seconds |
+
+Individual sources can override their cadence via the `polling_interval_seconds` field in their `config` JSONB column in the `sources` table. The `get_cadence_for_source()` function checks for this override first, falling back to the default if none is set, and enforces a minimum interval of 10 seconds.
+
+The Scheduler determines whether a source is due by calling `is_source_due()`, which considers several conditions. If a source has never run before (no entry in the `ingestion_runs` table), it is immediately due. If the last run failed, the Scheduler respects an exponential backoff computed by `compute_backoff()`: the delay starts at 60 seconds (`DEFAULT_BACKOFF_BASE`) and doubles with each retry up to a maximum of 3,600 seconds (`MAX_BACKOFF`). If a source has failed 10 consecutive times (`MAX_RETRY_COUNT`), the Scheduler stops scheduling it entirely until an operator manually resets the retry state. If the last run is still marked as `running`, the source is skipped to prevent double-scheduling. Otherwise, the Scheduler checks whether enough time has elapsed since the last completed run based on the source's cadence.
+
+Rate limiting adds another layer of protection. The `check_rate_limit()` function enforces two constraints. First, each source type has a per-type limit defined in `DEFAULT_RATE_LIMITS` — for example, `market_api` and `news_api` are each capped at 20 requests per minute, while `filings_api` and `macro_news` are capped at 10. Second, because `market_api` and `news_api` both use the same Polygon.io API key, a global Polygon rate limit of 45 requests per minute (`POLYGON_GLOBAL_RATE_LIMIT`) is enforced across both types combined. Rate limit state is tracked in Redis using keys of the form `stonks:ratelimit:{source_type}:{window}`, where the window is a minute-granularity timestamp. If a source type exceeds its limit, the Scheduler logs a warning and skips that source for the current tick.
+
+The Scheduler handles three categories of sources in each cycle. First, it fetches all active company-specific sources (excluding `macro_news`) by joining the `sources` and `companies` tables. Second, it fetches active macro news sources separately, since these may not have a `company_id`. Third, it fetches global market sources — those with `source_type='market_api'` and `company_id IS NULL` — which represent endpoints like the grouped daily bars that return data for all tickers in a single API call. For intraday bar sources, the Scheduler expands a single global source into per-ticker jobs for every active company.
+
+Each enqueued job payload includes the `source_id`, `company_id`, `ticker`, `legal_name`, `source_type`, `source_name`, `config`, `credibility_score`, a list of company `aliases` (fetched from the `company_aliases` table), and a `scheduled_at` timestamp. The job is pushed onto `stonks:queue:ingestion` via Redis `RPUSH`.
+
+Beyond scheduling, the Scheduler also performs periodic maintenance. Every ~20 cycles (~5 minutes), it runs `recover_stale_documents()` to re-enqueue documents that have been stuck in `parsed` status for longer than 240 minutes — a safety net for cases where Redis loses queue entries due to pod restarts or OOM events. Every ~40 cycles (~10 minutes), it runs `retry_failed_extractions()` to give documents in `extraction_failed` status another chance, resetting them to `parsed` and deleting the failed `document_intelligence` row so the Extractor treats them as fresh. Every ~100 cycles (~25 minutes), it runs `cleanup_all_tables()` to enforce retention policies across tables like `competitive_signal_records` (30 days), `ingestion_runs` (14 days), and `trading_decisions` (90 days).
+
+For more detail on the Scheduler's configuration and operational behavior, see the [Services Reference](../services.md).
+
+---
+
+## The Ingestion Worker: Adapter Dispatch and Persistence
+
+The Ingestion Worker (`services/ingestion/worker.py`) is a long-running process that continuously pops jobs from the `stonks:queue:ingestion` Redis list and processes them. On startup, it initializes one instance of each adapter class and stores them in a dispatch dictionary keyed by `source_type`:
+
+```
+adapters = {
+ "market_api": PolygonMarketAdapter(...),
+ "news_api": PolygonNewsAdapter(...),
+ "filings_api": SECEdgarAdapter(),
+ "web_scrape": WebScrapeAdapter(),
+ "broker": AlpacaBrokerAdapter(...),
+ "macro_news": MacroNewsAdapter(...),
+}
+```
+
+When a job arrives, the `process_job()` function looks up the appropriate adapter by `source_type` and calls its `fetch()` method with the ticker and source config. Before fetching, it records a new row in the `ingestion_runs` table with status `running`. If the adapter returns an error, the worker calls `record_retrieval_failure()` to update the run status and increment the source's retry counter with exponential backoff timing.
+
+On a successful fetch, the worker performs several steps in sequence. First, it uploads the raw payload to MinIO via `upload_raw_artifact()` in `services/shared/storage.py`. The target bucket is determined by the source type through the `SOURCE_BUCKET_MAP`: `market_api` payloads go to `stonks-raw-market`, `news_api` and `macro_news` payloads go to `stonks-raw-news`, and `filings_api` payloads go to `stonks-raw-filings`. Objects are stored under a path that encodes the source type, ticker, date hierarchy, and document ID — for example, `news_api/AAPL/2025/01/15/{run_id}/raw.json`.
+
+---
+
+## Content Deduplication via Redis
+
+After storing the raw artifact, the Ingestion Worker checks for duplicate content. Deduplication operates at two levels.
+
+At the payload level, the worker checks the overall `content_hash` (a SHA-256 digest of the raw API response) against Redis. The key pattern is `stonks:dedupe:{content_hash}` with a 24-hour TTL (86,400 seconds). If the hash is already present, the entire payload is skipped — the `ingestion_runs` row is marked as completed with `items_new=0`, and no downstream jobs are enqueued. If the hash is new, the worker sets the marker in Redis so future fetches of identical content are caught.
+
+At the individual item level, for source types other than `market_api` and `broker`, the worker calls `dedupe_items()` from `services/shared/dedupe.py`. This function checks each item against a layered deduplication strategy. The fast path checks Redis for both content-hash markers (`stonks:dedupe:{hash}`) and canonical-URL markers (`stonks:dedupe:url:{url_hash}`), both with 24-hour TTLs. If the Redis check misses, the function falls back to PostgreSQL, querying the `documents` table by `content_hash` or `canonical_url` for durable cross-source matching. When a duplicate is found through the PostgreSQL fallback, the function warms the Redis cache so subsequent checks are fast.
+
+Items identified as duplicates are not discarded entirely. If the duplicate document was originally ingested for a different company, the worker creates a cross-source mention link in the `document_company_mentions` table via `persist_document_company_mention()`. This ensures that a news article mentioning both Apple and Microsoft is linked to both companies even if it was first ingested through Apple's news source.
+
+New (non-duplicate) items are persisted to PostgreSQL through `persist_ingestion_items()` in `services/shared/metadata.py`, which inserts rows into the `documents` table and records company mentions in `document_company_mentions`. Each new document ID is then pushed onto `stonks:queue:parsing` for the Parser to process. After persistence, the worker calls `mark_as_seen()` to set Redis dedupe markers for both the content hash and canonical URL of each new item, ensuring that the next fetch cycle's deduplication checks are fast.
+
+On successful completion, the worker updates the `ingestion_runs` row with the final counts (`items_fetched`, `items_new`) and calls `reset_source_retry_state()` to clear any accumulated backoff from previous failures. For news-type sources (`news_api` and `macro_news`), the worker also updates the source's `config` JSONB column with the latest `published_utc` value, so the next fetch only retrieves newer articles.
+
+---
+
+## The Parser: Normalization, Quality Scoring, and Routing
+
+Documents that pass through ingestion arrive on the `stonks:queue:parsing` Redis list as JSON payloads containing a `document_id`, `ticker`, and `source_type`. The Parser Worker (`services/parser/worker.py`) pops these jobs and transforms raw HTML or text into normalized, quality-scored documents ready for AI extraction.
+
+The parsing pipeline begins with HTML fetching. If the document has a URL (looked up from the `documents` table if not present in the job payload), the worker calls `fetch_html()` to retrieve the page content. SEC EDGAR URLs receive a specialized `User-Agent` header to comply with the SEC's fair-access policy. The raw HTML is then passed to `parse_html()` in `services/parser/html_parser.py`, which runs a multi-stage extraction pipeline.
+
+The HTML parser first strips non-content tags — `script`, `style`, `nav`, `footer`, `header`, `aside`, `iframe`, and others — and removes boilerplate containers identified by CSS class or ID patterns (sidebars, ad slots, newsletter signups, social share bars, and similar UI elements). It then searches for the article body using a priority list of semantic selectors (`article`, `[role='main']`, `.article-body`, `.post-content`, and others). If no semantic match is found, it falls back to text-density scoring across candidate `div`, `section`, and `td` elements, selecting the block with the highest composite score based on text density, link density, paragraph count, and word count. The extracted text undergoes further cleaning: regex-based removal of residual boilerplate phrases (copyright notices, "subscribe to our newsletter" prompts, "share this article" fragments), removal of short orphan lines that are likely UI fragments, detection and collapse of repeated template blocks, and whitespace normalization.
+
+Metadata extraction pulls the document title (from `og:title` or ``), author, publisher (from `og:site_name` or hostname), publication date (from `article:published_time` or JSON-LD `datePublished`), canonical URL, language, description, and keywords from the HTML head elements.
+
+If the parsed body text is shorter than 500 characters, the worker attempts to enrich it by reading the raw API payload from MinIO and extracting the Polygon article description, keywords, and author fields for the matching article. This enrichment step ensures that even articles with minimal scrapeable HTML still have enough textual content for meaningful AI extraction.
+
+Quality scoring is performed by `score_parse_quality()` in `services/parser/html_parser.py`, which evaluates six weighted signals to produce a composite score between 0 and 0.95:
+
+| Signal | Weight | What It Measures |
+|--------------------|--------|-----------------------------------------------------------------|
+| `word_count` | 0.30 | Length of extracted text (thresholds at 20, 50, 150, 300 words) |
+| `body_found` | 0.20 | Whether a semantic article body element was located |
+| `diversity` | 0.15 | Vocabulary richness (unique words / total words) |
+| `sentence` | 0.15 | Presence of proper sentence structure (terminal punctuation) |
+| `paragraph` | 0.10 | Multi-paragraph structure (blocks separated by blank lines) |
+| `metadata` | 0.10 | Presence of title, author, publisher, and publication date |
+
+The composite score maps to a confidence label: scores below 0.35 are labeled `low`, scores between 0.35 and 0.65 are `medium`, and scores 0.65 and above are `high`. Documents with `low` confidence are marked with status `low_quality` in the `documents` table and are not enqueued for extraction — they are effectively filtered out of the pipeline at this stage.
+
+Company mention detection runs next. The worker fetches all known aliases from the `company_aliases` table (plus tickers and legal names from the `companies` table) and calls `detect_company_mentions()` in `services/parser/html_parser.py`. The matching strategy varies by alias length: one-to-two character aliases use case-sensitive word-boundary matching to avoid false positives (the letter "A" should not match every occurrence of the word "a"), three-to-four character aliases use case-insensitive word-boundary matching (standard ticker format), and aliases of five or more characters use case-insensitive substring matching (company names and brands). Confidence scores vary by alias type: ticker matches receive 0.9, legal name matches 0.85, general aliases 0.7, and brand matches 0.6. Multiple alias hits for the same company are deduplicated, keeping the highest-confidence match and summing match counts. Detected mentions are persisted to the `document_company_mentions` table.
+
+The normalized text and a structured parser output JSON (containing all metadata, quality signals, warnings, outbound links, tags, and mentions) are uploaded to the `stonks-normalized` MinIO bucket. The `documents` row is updated with the normalized storage reference, parser output reference, quality score, and confidence level.
+
+Finally, the Parser makes a routing decision. If the document's `document_type` is `macro_event`, it is pushed onto `stonks:queue:macro_classification` for the Global Event Classifier agent. All other documents are pushed onto `stonks:queue:extraction` for the Document Intelligence Extractor agent. Both queues feed into the Extractor service described in [Page 2](02-ai-agent-processing-and-extraction.md). The job payload includes the `document_id`, `ticker`, and the first 32,000 characters of the normalized text, giving the downstream agent immediate access to the content without needing to fetch it from MinIO.
+
+For additional detail on queue topology and data store layout, see the [Data Pipeline Architecture](../architecture-data-pipeline.md) documentation.
+
+---
+
+## What Comes Next
+
+At this point, raw data has been fetched from four external sources, deduplicated, stored in MinIO, parsed into normalized text, scored for quality, tagged with company mentions, and routed to the appropriate extraction queue. The documents sitting on `stonks:queue:extraction` and `stonks:queue:macro_classification` are clean, quality-filtered, and ready for AI processing. [Page 2 — AI Agent Processing and Structured Extraction](02-ai-agent-processing-and-extraction.md) picks up the story from here, explaining how the Document Intelligence Extractor and Global Event Classifier agents use LLM inference to transform these normalized documents into the structured JSON intelligence that feeds the rest of the pipeline.
diff --git a/docs/intelligence-pipeline-deep-dive/02-ai-agent-processing-and-extraction.md b/docs/intelligence-pipeline-deep-dive/02-ai-agent-processing-and-extraction.md
new file mode 100644
index 0000000..b0ddc34
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/02-ai-agent-processing-and-extraction.md
@@ -0,0 +1,164 @@
+# Page 2 — AI Agent Processing and Structured Extraction
+
+Documents that arrive on the `stonks:queue:extraction` and `stonks:queue:macro_classification` Redis queues are clean, quality-filtered, and normalized — but they are still unstructured text. The job of the Extractor service is to transform that text into structured JSON intelligence that the rest of the pipeline can reason about quantitatively. Two AI agents share this responsibility: the Document Intelligence Extractor handles company-specific news, filings, and transcripts, while the Global Event Classifier handles macro-level geopolitical and economic events. Both agents run through the same Ollama-based inference infrastructure, share a common JSON repair pipeline, and persist their results to PostgreSQL and MinIO for downstream consumption and audit.
+
+This page explains how each agent works, what schemas they produce, how the system validates and repairs LLM output, how runtime configuration is resolved from the database, and how the final structured records are persisted. For a visual overview of the full flow from ingestion through extraction, see the [Ingestion to Extraction Flow diagram](diagrams/ingestion-to-extraction-flow.md). For reference-level detail on agent configuration and the variant management API, see the [AI Agents Guide](../ai-agents.md).
+
+---
+
+## The Document Intelligence Extractor
+
+The Document Intelligence Extractor is the primary AI agent in the pipeline. Registered under the slug `document-extractor` in the `ai_agents` database table, it processes every non-macro document that passes through the Parser — news articles, SEC filings, earnings transcripts, and press releases. Its purpose is to read a normalized document and produce a structured JSON object that captures the document's summary, the companies it affects, the sentiment and impact for each company, the catalysts driving that impact, and the evidence supporting the analysis.
+
+The entry point is `services/extractor/main.py`, which runs a continuous worker loop polling the `stonks:queue:extraction` Redis list. When a job arrives, the worker extracts the `document_id`, `ticker`, and `text` fields from the JSON payload. If the job payload does not include the document text directly, the worker fetches it from MinIO using the `normalized_storage_ref` stored in the `documents` table — the Parser uploaded the normalized text to the `stonks-normalized` bucket during the previous pipeline stage (see [Page 1](01-data-ingestion-and-preparation.md)).
+
+The actual LLM inference is handled by `OllamaClient` in `services/extractor/client.py`. The client sends the document to a local Ollama instance via the `/api/chat` HTTP endpoint with `stream=False` and `think=False`. The `think=False` flag is a deliberate performance choice — it disables the model's chain-of-thought reasoning phase, which would otherwise add two to four minutes of latency per document. The client does not use Ollama's `format` parameter for structured output because of a known Ollama bug (#14645) where the format constraint is silently ignored when `think=False` on qwen3.5 models. Instead, the system relies on prompt engineering to produce JSON and repairs any syntax issues after the fact.
+
+The prompt sent to the model has two parts. The system prompt, defined in `services/extractor/prompts.py`, establishes the model's role as a financial document analyst and sets strict output rules: return only a single JSON object, no markdown fences, no explanation text, every schema field is required, use `"other"` for `catalyst_type` when unsure, keep evidence spans under 20 words, and limit key facts to three to five items. The user prompt, built by `build_extraction_prompt()` in the same module, provides the document text along with document-type-specific guidance. Four guidance variants exist — one each for articles, filings, transcripts, and press releases — each calibrated to the conventions and biases of that document type. For example, the filing guidance instructs the model to preserve the precise legal language of SEC documents, while the press release guidance warns that sentiment may be biased positive and directs the model to focus on concrete metrics rather than marketing language.
+
+The user prompt also includes a list of all tracked tickers from the `companies` table, along with rules for how the model should use them. If a tracked ticker appears verbatim in the text, the model must include it in the output with at least one evidence span. If the article discusses a sector or theme that clearly affects a tracked company (oil prices affecting XOM, AI chip demand affecting NVDA), the model should include that company as well. The model is explicitly told not to invent tickers that are not in the provided list. Documents longer than 8,000 characters are truncated before being included in the prompt, with a `[... truncated for extraction ...]` marker appended.
+
+The `OllamaClient` also supports a `context_window` override via the Ollama `num_ctx` option, which can be configured per agent variant through the `AgentConfigResolver` mechanism described later in this page.
+
+---
+
+## The ExtractionResult Schema
+
+The structured output that the Document Intelligence Extractor produces is defined by the `ExtractionResult` Pydantic model in `services/extractor/schemas.py`. Every field is required — the model has no defaults — so the generated JSON schema forces the LLM to produce every field explicitly. The top-level fields are:
+
+**`summary`** — a concise one-to-three sentence summary of the document's main point. This becomes the human-readable description stored in the `document_intelligence` table.
+
+**`companies`** — an array of `CompanyExtractionItem` objects, one per affected company. Each company entry contains:
+
+- `ticker` — the stock ticker symbol (validated against a regex pattern of one to five uppercase letters).
+- `company_name` — the full company name as referenced in the document.
+- `relevance` — a float between 0.0 and 1.0 indicating how relevant the document is to this company, where 0 means tangential and 1 means the company is the primary subject.
+- `sentiment` — one of `positive`, `negative`, `neutral`, or `mixed`, representing the overall sentiment toward this company in the document.
+- `impact_score` — a float between 0.0 and 1.0 estimating the magnitude of impact, where 0 is negligible and 1 is highly material.
+- `impact_horizon` — one of `intraday`, `1d`, `1d_7d`, `1d_30d`, `30d_90d`, or `90d_plus`, indicating the expected timeframe over which the impact will play out.
+- `catalyst_type` — exactly one of `earnings`, `product`, `legal`, `macro`, `supply_chain`, `m_and_a`, `rating_change`, or `other`. The prompt instructs the model to use `other` when none of the specific categories fit.
+- `key_facts` — a list of facts explicitly stated in the document. The prompt emphasizes that the model must not infer or fabricate facts.
+- `risks` — a list of risks explicitly mentioned in the document.
+- `evidence_spans` — short verbatim quotes from the document supporting the analysis. The prompt requests these be kept under 20 words each.
+
+**`macro_themes`** — a list of broad economic or market themes mentioned in the document, such as `rates`, `inflation`, or `ai_capex`.
+
+**`novelty_score`** — a float between 0.0 and 1.0 indicating how novel or surprising the information is. Routine earnings reports score low; unexpected regulatory actions score high. This value feeds into the novelty bonus component of the signal weighting formula described in [Page 3](03-signal-scoring-and-weighted-signals.md).
+
+**`confidence`** — a float between 0.0 and 1.0 representing the model's confidence in the accuracy of its extraction. Lower values indicate ambiguous or incomplete source text. This value becomes the confidence gate input for signal scoring.
+
+**`extraction_warnings`** — a list of issues encountered during extraction, such as `ambiguous_ticker`, `incomplete_text`, or `low_confidence`. These warnings are persisted alongside the intelligence record for operational monitoring.
+
+The JSON schema is generated programmatically from the Pydantic models via `generate_json_schema()` in `services/extractor/schemas.py`, which calls Pydantic's `model_json_schema()` and then inlines all `$defs` references so the schema is self-contained and Ollama-friendly.
+
+---
+
+## The Global Event Classifier
+
+Not all documents describe company-specific developments. Macro news articles — those tagged with `document_type='macro_event'` by the Parser — describe events that affect entire markets, sectors, or economies: trade wars, central bank rate decisions, commodity supply disruptions, geopolitical conflicts. These documents are routed to the `stonks:queue:macro_classification` Redis queue and processed by the Global Event Classifier agent, registered under the slug `event-classifier` in the `ai_agents` table.
+
+The classifier is implemented in `services/extractor/event_classifier.py`. When the extractor worker in `services/extractor/main.py` pops a job and determines that the document type is `macro_event` (either because the job came from the macro queue or because the `documents` table records it as such), it routes the document to `_process_macro_classification()` instead of the standard extraction pipeline. This function calls `classify_global_event()`, which builds a dedicated prompt, sends it to Ollama through the same `OllamaClient` infrastructure, parses the response, and persists the result.
+
+The classifier's system prompt is distinct from the extractor's. It establishes the model's role as a macro-level news classifier and includes explicit anti-hallucination rules that are critical to preventing the classifier from overreaching. The prompt states that the model should only classify articles about macro events that affect entire markets, sectors, or economies — trade wars, interest rate changes, commodity supply disruptions, regulatory changes, geopolitical conflicts, natural disasters. It explicitly lists what should not be classified as macro events: individual company earnings, lawsuits against a single company, single-company management changes, individual stock analysis, company-specific debt or bankruptcy, and product launches by one company. For these company-specific articles that were incorrectly routed, the model is instructed to set severity to `"low"`, confidence below 0.3, and leave the `affected_regions`, `affected_sectors`, and `affected_commodities` arrays empty.
+
+The user prompt, built by `build_event_classification_prompt()`, reinforces these anti-hallucination rules and provides additional guidance. It instructs the model to only extract facts explicitly stated in the text, to set confidence below 0.4 for vague or speculative content, to distinguish announced policy from rumored policy, and to reserve `"critical"` severity for events affecting multiple countries or entire global markets. Articles longer than 6,000 characters are truncated before inclusion in the prompt.
+
+The output schema is the `GlobalEvent` dataclass, which contains:
+
+- `event_types` — a list of impact type strings, drawn from a fixed set: `supply_disruption`, `demand_shift`, `cost_increase`, `regulatory_pressure`, `currency_impact`, `commodity_shock`, `trade_barrier`, and `geopolitical_risk`. The model is instructed to include all applicable types rather than collapsing to a single category.
+- `severity` — one of `low`, `moderate`, `high`, or `critical`.
+- `affected_regions` — ISO 3166-1 alpha-2 country codes or region names (e.g., `US`, `CN`, `EU`, `GB`, `JP`). Only regions explicitly mentioned or clearly implied should be included.
+- `affected_sectors` — GICS sector identifiers such as `Energy`, `Financials`, `Information Technology`, or `Industrials`.
+- `affected_commodities` — commodity identifiers like `crude_oil`, `natural_gas`, `gold`, `copper`, `wheat`, `lithium`, or `semiconductors`. An empty list if no commodities are directly affected.
+- `summary` — a one-to-three sentence summary of the event and its market implications.
+- `key_facts` — facts explicitly stated in the article, limited to three to five items.
+- `estimated_duration` — one of `short_term` (days to weeks), `medium_term` (weeks to months), or `long_term` (months to years).
+- `confidence` — a float between 0.0 and 1.0, clamped during parsing.
+
+Each `GlobalEvent` also carries a `model_metadata` object recording the provider (`ollama`), model name, prompt version (`event-classification-v1`), and schema version (`1.0.0`), plus a `source_document_id` linking back to the originating document.
+
+After a successful classification, the system computes macro impact records for all tracked companies using the exposure-based interpolation engine in `services/aggregation/interpolation.py`. Each company's exposure profile — geographic revenue mix, supply chain regions, key input commodities, regulatory jurisdictions, and market position tier — determines how much a given macro event affects that company. Companies with non-zero macro impact scores get `macro_impact_records` rows persisted to PostgreSQL, and aggregation jobs are enqueued to `stonks:queue:aggregation` for each affected ticker. The extractor worker tracks consecutive macro classification failures and emits a critical-level alert after three consecutive failures, continuing with company-only signals in the meantime.
+
+---
+
+## The JSON Repair Pipeline
+
+LLM output is inherently unreliable at the syntactic level. Models sometimes wrap JSON in markdown fences, produce trailing commas, leave strings unterminated, or truncate output mid-object when they hit token limits. The extractor addresses this with a three-stage JSON repair pipeline implemented across `services/extractor/client.py` and `services/extractor/schemas.py`.
+
+The first stage is a direct `json.loads()` call. If the raw model output is already valid JSON, no repair is needed and the pipeline moves straight to validation. This is the fast path for well-behaved model responses.
+
+The second stage strips markdown fences. Models frequently wrap their output in `` ```json ... ``` `` blocks despite being told not to. The `_strip_markdown_fences()` function in `services/extractor/client.py` uses a regex to detect and remove these wrappers before attempting another parse.
+
+The third stage invokes the `json-repair` library as a fallback. The `_repair_json()` function in `services/extractor/client.py` calls `repair_json()` with `return_objects=False` to get a repaired JSON string. This library handles a wide range of common LLM JSON errors — trailing commas, missing quotes, unescaped characters — that would otherwise require custom repair logic.
+
+The `services/extractor/schemas.py` module contains an additional layer of repair logic in its own `_repair_json()` function, which handles cases that the library might miss. It strips non-JSON prefixes (models sometimes prepend explanatory text before the opening brace), removes control characters that break parsing, fixes trailing commas before closing brackets, and as a last resort calls `_repair_truncated_json()` — a state-machine parser that walks the string tracking bracket depth and string state, then appends the necessary closing tokens to complete a truncated JSON object.
+
+For the Global Event Classifier, the `_parse_classification_response()` function in `services/extractor/event_classifier.py` reuses the same `_strip_markdown_fences()` and `_repair_json()` functions from the client module, and additionally handles the case where the model wraps the output object in a single-element list — a quirk observed with some model configurations.
+
+---
+
+## Structural and Semantic Validation
+
+Repairing JSON syntax is only the first step. The `validate_extraction()` function in `services/extractor/schemas.py` performs both structural and semantic validation on the parsed output, and the distinction between the two is important for understanding the retry logic.
+
+Structural validation begins with normalization. The `_normalize_extraction_data()` function fills in missing top-level fields with sensible defaults (empty summary, empty companies array, 0.5 novelty score, 0.3 confidence), clamps numeric fields to the [0.0, 1.0] range, and normalizes per-company fields. Catalyst types that the model produces as free-text alternatives — `"strategic pivot"`, `"acquisition"`, `"lawsuit"`, `"inflation"`, `"launch"` — are mapped to their canonical enum values through a comprehensive alias dictionary. Impact horizons like `"long-term"`, `"short"`, `"immediate"`, or `"near-term"` are similarly mapped to the valid set (`intraday`, `1d`, `1d_7d`, `1d_30d`, `30d_90d`, `90d_plus`). After normalization, the data is validated against the `ExtractionResult` Pydantic model, which enforces type constraints, enum membership, and range bounds.
+
+Semantic validation catches issues that are structurally valid but logically suspect. The `_semantic_checks()` function runs a series of cross-field consistency checks that produce either errors (which trigger a retry) or warnings (which are logged but do not block acceptance). Semantic errors include duplicate tickers across company entries, missing ticker fields, and invalid impact horizon values. Semantic warnings include empty summaries, low confidence with companies present, invalid ticker formats (not matching the one-to-five uppercase letter pattern), missing evidence spans, evidence spans that are too short (under 8 characters) or too long (over 500 characters), high impact scores with no supporting key facts, very low relevance scores, and strong sentiment paired with negligible impact scores.
+
+When the original document text is available, the validator also performs an evidence grounding check: each evidence span is searched for in the source text (case-insensitive), and spans not found in the document are flagged with a warning. This helps detect hallucinated evidence — quotes the model fabricated rather than extracted from the actual text.
+
+If validation produces any semantic errors, the `ValidationReport` is marked as invalid and the `OllamaClient` retry loop treats it as a failed attempt. The retry logic uses exponential backoff with configurable parameters: a base delay (default from `OllamaConfig`), a multiplier applied on each retry, and a maximum delay cap. The number of retries is configurable per agent through the `max_retries` field in the `ai_agents` or `agent_variants` table. Non-retryable errors — HTTP 400, 401, 403, 404, and 422 responses from Ollama — short-circuit the retry loop immediately, since these indicate a problem with the request itself rather than a transient model failure.
+
+Every attempt, whether successful or not, is recorded in an `ExtractionAttempt` dataclass that captures the raw output, validation report, error description, duration in milliseconds, model name, and whether the error was retryable. The full list of attempts is preserved in the `ExtractionResponse` for audit purposes and uploaded to MinIO by the persistence layer.
+
+---
+
+## The AgentConfigResolver: Hot-Swapping Models and Prompts
+
+Both the Document Intelligence Extractor and the Global Event Classifier resolve their runtime configuration through the `AgentConfigResolver` in `services/shared/agent_config.py`. This mechanism allows operators to change models, prompts, timeouts, retry counts, and token budgets without restarting any service — changes take effect within 60 seconds.
+
+The resolver works by querying the `ai_agents` and `agent_variants` PostgreSQL tables with a single SQL statement that uses `COALESCE` to prefer variant values over base agent values. When the extractor worker starts, it creates an `AgentConfigResolver` instance with a 60-second TTL cache and calls `resolver.resolve("document-extractor")` to get the active configuration. If an active variant exists for the agent (enforced by a unique partial index on `agent_variants` that allows at most one active variant per agent), the variant's `model_name`, `system_prompt`, `temperature`, `max_tokens`, `context_window`, `timeout_seconds`, and `max_retries` override the base agent's values wherever the variant provides a non-NULL value. If no active variant exists, the base agent's configuration is used. If the database query fails entirely, the resolver returns `None` and the worker falls back to environment-variable-based `OllamaConfig` defaults.
+
+The resolved configuration is captured in a `ResolvedAgentConfig` frozen dataclass that includes the `agent_id`, `variant_id` (if any), `model_provider`, `model_name`, `system_prompt`, `user_prompt_template`, `prompt_version`, `temperature`, `max_tokens`, `context_window`, `input_token_limit`, `token_budget`, `timeout_seconds`, and `max_retries`. The extractor worker uses this to build an `OllamaConfig` that is passed to the `OllamaClient`.
+
+The 60-second TTL cache means the resolver only hits the database once per minute per agent slug. Cache entries are keyed by slug and timestamped with `time.monotonic()`. When a cached entry expires, the next `resolve()` call re-queries the database and refreshes the cache. The `invalidate()` method can clear a single slug or the entire cache, though in practice the TTL-based expiry is sufficient for normal operations.
+
+The extractor worker re-resolves its configuration every 100 jobs. If the resolved model name has changed (for example, because an operator activated a variant that uses a different model), the worker closes the old `OllamaClient` and creates a new one with the updated configuration. The event classifier is resolved separately and can use a different model than the document extractor — the worker maintains two independent `OllamaClient` instances when the models differ.
+
+Token budget enforcement adds another layer of control. If a variant specifies a `token_budget` (total tokens per hour), the worker checks the `agent_performance_log` table before each invocation to see whether the budget has been exceeded. If so, the invocation is skipped entirely. Input token limits work similarly: if a variant sets an `input_token_limit`, the worker truncates the document text to approximately that many tokens (estimated at four characters per token) before sending it to the model.
+
+For a complete guide to creating variants, activating them, and comparing their performance, see the [AI Agents Guide](../ai-agents.md).
+
+---
+
+## Persistence: From Extraction to Database
+
+Once the LLM produces a valid extraction and it passes validation, the `persist_extraction()` function in `services/extractor/worker.py` orchestrates the full persistence pipeline. This function writes to both MinIO (for audit) and PostgreSQL (for downstream consumption), ensuring that every extraction attempt is fully traceable.
+
+The MinIO persistence layer uploads four artifacts per extraction, all stored under date-partitioned paths in dedicated buckets. The prompt metadata (prompt version, schema version, model name) goes to `stonks-llm-prompts`. The raw model output for every attempt — including failed ones — goes to `stonks-llm-results`, preserving the full retry history. A validation report summarizing the final attempt's status, errors, and warnings is uploaded alongside the raw output. On success, the final parsed intelligence object (the `ExtractionResult` serialized as JSON) is uploaded to a separate path for easy retrieval.
+
+The PostgreSQL persistence writes to two tables. The `document_intelligence` table receives one row per document, containing the summary, macro themes, novelty score, source credibility, extraction warnings, confidence, model metadata (provider, model name, prompt version, schema version), references to the MinIO artifacts (raw output ref, prompt ref), validation status (`valid` or `failed`), validation errors, and retry count. This row is the authoritative record of what the AI extracted from the document.
+
+The `document_impact_records` table receives one row per company mention within the extraction. Each impact record is linked to the parent `document_intelligence` row via `intelligence_id` and to the `companies` table via `company_id`. The record captures the ticker, relevance, sentiment, impact score, impact horizon, catalyst type, key facts, risks, and evidence spans for that specific company. The `company_id` is resolved from a ticker-to-UUID mapping that the worker maintains by querying the `companies` table (refreshed every 100 jobs). If a ticker in the extraction output does not match any tracked company, the impact record is skipped with a warning — the system only persists impact records for companies in its tracked universe.
+
+After persisting the intelligence and impact records, the worker updates the document's status in the `documents` table to `extracted` (or `extraction_failed` if all retry attempts were exhausted). Even failed extractions get a `document_intelligence` row with `validation_status='failed'`, empty summary, zero confidence, and the accumulated error messages — this ensures the failure is visible in the database rather than silently lost.
+
+Performance metrics are collected for every extraction via `collect_metrics()` in `services/extractor/metrics.py` and persisted to a metrics table. Prometheus counters and histograms track extraction attempts, duration, retries, confidence distribution, validation errors, and estimated token usage (input and output, estimated at four characters per token). When a resolved agent config is available, the worker also logs to the `agent_performance_log` table with variant attribution, enabling the A/B comparison queries described in the [AI Agents Guide](../ai-agents.md).
+
+For the Global Event Classifier, persistence follows a parallel path. The prompt and raw output are uploaded to MinIO under an `event_classification/macro/` path prefix. The parsed `GlobalEvent` is persisted to the `global_events` PostgreSQL table, which stores the event types, severity, affected regions, affected sectors, affected commodities, summary, key facts, estimated duration, confidence, source document ID, and model metadata. Downstream, the macro interpolation engine computes `macro_impact_records` for each affected company and persists those as well.
+
+---
+
+## Enqueuing Aggregation Jobs
+
+The final step in the extraction pipeline is to notify the downstream aggregation engine that new intelligence is available. After a successful document extraction, the worker pushes a job onto the `stonks:queue:aggregation` Redis list containing the ticker of the affected company. The aggregation engine (described in [Page 3](03-signal-scoring-and-weighted-signals.md)) will pick up this job and recompute the weighted signals and trend summaries for that ticker, incorporating the freshly extracted intelligence.
+
+For macro events, the enqueue logic is more expansive. After the Global Event Classifier produces a `GlobalEvent` and the interpolation engine computes macro impact records, the worker enqueues an aggregation job for every ticker that received a non-zero macro impact score. A single macro event — say, a new tariff announcement affecting the Energy and Industrials sectors — can trigger aggregation recomputation for dozens of tickers simultaneously. The aggregation job payload includes both the `ticker` and the `macro_event_id`, so the aggregation engine knows to incorporate the new macro signals.
+
+The worker alternates between the extraction and macro classification queues to prevent starvation: every third job is pulled from `stonks:queue:macro_classification`, with the remaining two-thirds from `stonks:queue:extraction`. If the preferred queue is empty, the worker falls back to the other queue, ensuring that neither pipeline stalls while the other has work available.
+
+---
+
+## What Comes Next
+
+At this point, documents have been transformed from unstructured text into structured JSON intelligence — `ExtractionResult` objects for company-specific documents and `GlobalEvent` objects for macro news. These structured records are persisted in PostgreSQL and their tickers have been enqueued for aggregation. But raw extraction output is not yet actionable for trading decisions. The extraction tells us that a document is bearish for AAPL with an impact score of 0.7 and a confidence of 0.8, but it does not tell us how much weight that signal should carry relative to other signals about AAPL, or how it compares to signals from different sources, time periods, or market conditions. [Page 3 — Signal Scoring and the WeightedSignal Abstraction](03-signal-scoring-and-weighted-signals.md) picks up the story from here, explaining how the aggregation engine transforms these raw extraction outputs into weighted signals through confidence gating, recency decay, source credibility scoring, novelty bonuses, and market context multipliers.
diff --git a/docs/intelligence-pipeline-deep-dive/03-signal-scoring-and-weighted-signals.md b/docs/intelligence-pipeline-deep-dive/03-signal-scoring-and-weighted-signals.md
new file mode 100644
index 0000000..fd00ee6
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/03-signal-scoring-and-weighted-signals.md
@@ -0,0 +1,210 @@
+# Page 3 — Signal Scoring and the WeightedSignal Abstraction
+
+The extraction pipeline described in [Page 2](02-ai-agent-processing-and-extraction.md) produces structured intelligence records — `document_impact_records` for company-specific documents, `macro_impact_records` for global events, and `competitive_signal_records` for cross-company pattern propagation. Each record carries a sentiment, an impact score, a confidence value, and a publication timestamp. But these raw values are not directly comparable. A high-confidence extraction from a reputable source published ten minutes ago should carry far more weight than a low-confidence extraction from an unknown source published three weeks ago. A document that breaks genuinely novel information should matter more than one that rehashes yesterday's earnings call. And when the market is moving fast — high volatility, surging volume — fresh signals become even more critical.
+
+The signal scoring layer in `services/aggregation/scoring.py` solves this problem by transforming each raw intelligence record into a `WeightedSignal` object: a document reference paired with a composite aggregation weight that encodes recency, credibility, novelty, confidence, and market conditions into a single number. This page explains how that weight is computed, how sentiment labels become numeric values, and how three independent signal layers — Company, Macro, and Competitive — each produce `WeightedSignal` objects that are concatenated into a unified list before the aggregation engine computes trend summaries. For a visual breakdown of the composite weight formula, see the [Weighted Signal Computation diagram](diagrams/weighted-signal-computation.md). For the full picture of how the three layers merge, see the [Three-Layer Signal Merging diagram](diagrams/three-layer-signal-merging.md).
+
+---
+
+## The WeightedSignal and SignalWeight Dataclasses
+
+The core abstraction is the `WeightedSignal` dataclass, defined in `services/aggregation/scoring.py`. It pairs a document reference with the computed weight and the signal's sentiment and impact values:
+
+- **`document_id`** — the UUID of the source document (for company and macro signals) or a synthetic identifier for pattern-derived signals (e.g., `pattern:AAPL:earnings:7d`).
+- **`weight`** — a `SignalWeight` object containing the component breakdown and the final combined score.
+- **`sentiment_value`** — a numeric sentiment value: `+1.0` for positive, `-1.0` for negative, `0.0` for neutral or mixed.
+- **`impact_score`** — the magnitude of impact, drawn from the extraction's per-company impact score for company signals, or scaled by a layer-specific weight multiplier for macro and competitive signals.
+
+The `SignalWeight` dataclass captures the individual components that feed into the combined weight, making the scoring decision fully transparent and auditable:
+
+- **`recency`** — the exponential decay weight based on document age.
+- **`credibility`** — the source credibility weight after clamping and exponentiation.
+- **`novelty_bonus`** — the additive bonus derived from the document's novelty score.
+- **`confidence_gate`** — either `1.0` (signal passes) or `0.0` (signal is gated out).
+- **`market_ctx_multiplier`** — a multiplicative boost from market conditions, always `>= 1.0`.
+- **`combined`** — the final composite weight used by the aggregation engine.
+
+The `ScoringConfig` frozen dataclass holds all tunable parameters for the scoring functions — half-life hours per window, credibility bounds, novelty bonus cap, confidence floor, and market context thresholds. A module-level `DEFAULT_CONFIG` singleton provides the production defaults, but every scoring function accepts an optional `config` parameter so that tests and alternative configurations can override any parameter without modifying global state.
+
+---
+
+## The Composite Weight Formula
+
+The `compute_signal_weight()` function in `services/aggregation/scoring.py` computes the combined weight for a single document signal. The formula is:
+
+```
+combined = gate × recency × credibility × (1 + novelty_bonus) × market_context_multiplier
+```
+
+Each factor is computed independently and then multiplied together. This multiplicative structure means that any single factor can zero out the entire weight (the confidence gate) or amplify it (the market context multiplier), and the interaction between factors is naturally captured — a highly credible, very recent document with novel information in a volatile market receives the maximum possible weight, while a stale, low-credibility document with routine information receives a weight close to zero.
+
+The following sections describe each component in detail.
+
+---
+
+## Confidence Gate
+
+The confidence gate is the first and most decisive filter. If the extraction confidence for a document falls below the `confidence_floor` threshold — set to `0.2` in the default `ScoringConfig` — the gate evaluates to `0.0` and the entire combined weight becomes zero. The document is effectively excluded from aggregation. If the confidence meets or exceeds the threshold, the gate evaluates to `1.0` and has no further effect on the weight.
+
+This binary gate exists because documents with very low extraction confidence are too unreliable to aggregate. A confidence of 0.15 typically means the LLM struggled to parse the document — perhaps the text was truncated, the language was ambiguous, or the document type was unusual. Including such signals would add noise rather than information. The threshold of 0.2 is deliberately low; it filters only the most unreliable extractions while allowing moderately confident signals to participate (their lower confidence is reflected through the credibility component instead).
+
+---
+
+## Recency Decay
+
+The `recency_weight()` function computes an exponential decay based on how old a document is relative to the aggregation anchor time. The formula is:
+
+```
+w = 2^(−age_hours / half_life)
+```
+
+A document published exactly one half-life ago receives a recency weight of `0.5`. A document published two half-lives ago receives `0.25`, and so on. A document published at or after the reference time receives the maximum weight of `1.0`.
+
+The half-life varies by trend window, reflecting the intuition that shorter windows need faster decay to stay responsive, while longer windows should give older documents more influence. The default half-lives, configured in `ScoringConfig.half_life_hours`, are:
+
+| Window | Half-Life |
+|--------|-----------|
+| `intraday` | 2 hours |
+| `1d` | 12 hours |
+| `7d` | 72 hours (3 days) |
+| `30d` | 240 hours (10 days) |
+| `90d` | 720 hours (30 days) |
+
+For the intraday window, a document published four hours ago already has a recency weight of `0.25` — it is rapidly losing influence as newer information arrives. For the 90-day window, that same four-hour-old document still has a recency weight of essentially `1.0`, because the 30-day half-life means age only becomes significant over weeks.
+
+A floor value of `min_recency_weight = 0.01` prevents very old documents from being completely zeroed out. Even a document from months ago retains a trace-level weight of 1%, ensuring it can still contribute to trend computation if no newer signals exist. Both timestamps are normalized to UTC; naive datetimes are treated as UTC to avoid timezone-related scoring errors.
+
+---
+
+## Source Credibility
+
+The `credibility_weight()` function transforms a source's credibility score into a weight component. The raw credibility value — a float between 0.0 and 1.0 stored in the `document_intelligence` table — is first clamped to the range `[0.1, 1.0]` using the `credibility_floor` and `credibility_ceiling` parameters from `ScoringConfig`. This clamping ensures that even the least credible sources retain a minimum weight of 0.1 rather than being completely silenced, while preventing any source from exceeding a weight of 1.0.
+
+After clamping, the value is raised to the `credibility_exponent` power. The default exponent is `1.0`, which means the clamped credibility passes through unchanged. Setting the exponent above 1.0 would penalize low-credibility sources more aggressively — for example, an exponent of 2.0 would reduce a credibility of 0.5 to a weight of 0.25. Setting it below 1.0 would flatten the curve, making the system more tolerant of lower-credibility sources. The exponent is configurable through `ScoringConfig` to allow operators to tune the credibility sensitivity without changing the scoring code.
+
+---
+
+## Novelty Bonus
+
+The novelty bonus rewards documents that contain genuinely new information. The bonus is computed as:
+
+```
+novelty_bonus = novelty_score × novelty_bonus_max
+```
+
+where `novelty_score` is the 0.0-to-1.0 value produced by the extraction model (see the `ExtractionResult` schema in [Page 2](02-ai-agent-processing-and-extraction.md)) and `novelty_bonus_max` is `0.25` by default. This means the bonus ranges from `0.0` (completely routine information) to `0.25` (maximally novel information), providing up to a 25% boost to the signal weight.
+
+The bonus enters the composite formula as `(1 + novelty_bonus)`, so it acts as a multiplicative amplifier on the base weight. A document with a novelty score of 1.0 gets its weight multiplied by 1.25; a document with a novelty score of 0.0 gets multiplied by 1.0 (no change). This design ensures that novelty can only increase a signal's weight, never decrease it — routine information is not penalized, it simply does not receive the bonus.
+
+---
+
+## Market Context Multiplier
+
+The `market_context_multiplier()` function computes a boost factor based on real-time market conditions for the ticker being aggregated. The multiplier is always `>= 1.0`, meaning market context can only amplify signal weights, never reduce them. When no market context data is available (the `MarketContext` object from `services/shared/schemas.py` has `has_data == False`), the multiplier defaults to `1.0`.
+
+Two market features contribute to the boost:
+
+**Volatility boost.** When the ticker's price volatility exceeds the `volatility_recency_boost_threshold` (default `1.0` in price units), the excess volatility is transformed through a logarithmic scaling function: `log₁₊(excess) × 0.15`. The logarithmic scaling prevents extreme volatility from producing runaway weight amplification. The boost is capped at `volatility_recency_boost_max = 0.30`, so the maximum volatility contribution is a 30% weight increase. The rationale is that in highly volatile markets, fresh intelligence is disproportionately valuable — a signal about NVDA matters more when NVDA is swinging 5% intraday than when it is trading in a tight range.
+
+**Volume surge boost.** When the ticker's volume change percentage exceeds `volume_surge_threshold_pct = 50.0%` (meaning trading volume is at least 50% above the prior period's average), a flat `volume_surge_boost = 0.15` is added. Unlike the volatility boost, this is binary — either the volume threshold is met and the full 15% boost applies, or it is not and no boost is added. High-volume moves carry more conviction because they represent broader market participation rather than thin-market noise.
+
+The two boosts are additive within the multiplier: `multiplier = 1.0 + volatility_boost + volume_surge_boost`. In the most extreme case — high volatility and a volume surge — the combined multiplier reaches `1.0 + 0.30 + 0.15 = 1.45`, amplifying the signal weight by 45%. The `MarketContext` data is fetched by `services/aggregation/market_context.py` from the market data tables in PostgreSQL, using the same ticker and window parameters as the impact record query.
+
+---
+
+## Sentiment Mapping
+
+Before signals can be aggregated into trend summaries, the categorical sentiment labels from the extraction output must be converted to numeric values. The `sentiment_to_numeric()` function in `services/aggregation/scoring.py` performs this mapping:
+
+| Sentiment Label | Numeric Value |
+|----------------|---------------|
+| `positive` | `+1.0` |
+| `negative` | `-1.0` |
+| `neutral` | `0.0` |
+| `mixed` | `0.0` |
+
+The mapping is case-insensitive. Any unrecognized label defaults to `0.0`. The choice to map both `neutral` and `mixed` to `0.0` is deliberate — a mixed-sentiment document (one that contains both positive and negative signals for the same company) should not push the trend in either direction. The contradiction between the positive and negative aspects is captured separately by the contradiction detection system described in [Page 4](04-trend-aggregation-and-accumulating-signals.md), rather than being baked into the sentiment value itself.
+
+For macro signals, the direction-to-sentiment mapping in `services/aggregation/worker.py` follows the same pattern: `positive` maps to `+1.0`, `negative` to `-1.0`, and both `mixed` and `neutral` to `0.0`. For competitive signals built by `build_pattern_weighted_signals()` in `services/aggregation/signal_propagation.py`, the sentiment is derived from the pattern's directional bias: `+1.0` if `bullish_pct > bearish_pct`, `-1.0` otherwise.
+
+---
+
+## Weighted Sentiment Average
+
+The `weighted_sentiment_average()` function computes the central metric that drives trend direction: a weight-adjusted average sentiment across all signals for a ticker in a given window. The formula is:
+
+```
+weighted_avg = Σ(combined_weight × impact_score × sentiment_value) / Σ(combined_weight × impact_score)
+```
+
+Each signal contributes its sentiment value scaled by both its composite weight and its impact score. The denominator normalizes by the total effective weight, producing a value in the range `[-1.0, +1.0]`. A result near `+1.0` means the weighted evidence is overwhelmingly positive; near `-1.0` means overwhelmingly negative; near `0.0` means either neutral or evenly split.
+
+The use of `combined_weight × impact_score` as the effective weight means that high-impact, high-weight signals dominate the average. A single high-confidence, recent, credible document with a strong impact score can outweigh several older, lower-impact documents — which is the intended behavior. The aggregation engine in `services/aggregation/worker.py` passes this weighted average to `derive_trend_direction()`, which maps it to a `TrendDirection` enum value (bullish, bearish, mixed, or neutral) using the thresholds described in [Page 4](04-trend-aggregation-and-accumulating-signals.md).
+
+If the total effective weight is zero — either because no signals exist or all signals were gated out by the confidence floor — the function returns `0.0`, which maps to a neutral trend direction.
+
+---
+
+## The Three Signal Layers
+
+The aggregation engine in `services/aggregation/worker.py` does not treat all intelligence sources equally. Signals flow through three independent layers, each with a different relative weight, before being concatenated into a single `WeightedSignal` list for trend computation. This layered architecture allows the system to incorporate diverse intelligence sources while controlling how much influence each source type has on the final trend.
+
+### Layer 1 — Company Signals (Weight: 1.0)
+
+Company signals are the primary layer. They are built by `build_weighted_signals()` in `services/aggregation/worker.py` from `document_impact_records` — the per-company extraction output produced by the Document Intelligence Extractor (see [Page 2](02-ai-agent-processing-and-extraction.md)). Each impact record's sentiment is converted via `sentiment_to_numeric()`, and its impact score is used directly without any layer-level scaling. The `compute_signal_weight()` function produces the composite weight using the document's publication time, source credibility, novelty score, extraction confidence, and the ticker's current market context.
+
+Company signals carry a relative weight of `1.0` — they are the baseline against which other layers are measured. This reflects the design principle that direct, company-specific intelligence (an earnings report about AAPL, a product launch by TSLA, a lawsuit against META) is the most relevant and reliable signal for that company's trend.
+
+### Layer 2 — Macro Signals (Weight: 0.3)
+
+Macro signals capture the indirect impact of global events on individual companies. They are built by `build_macro_weighted_signals()` in `services/aggregation/worker.py` from `macro_impact_records` — the per-company impact scores computed by the exposure-based interpolation engine after the Global Event Classifier processes a macro news article. The sentiment is mapped from the `impact_direction` field (`positive` → `+1.0`, `negative` → `-1.0`, `mixed`/`neutral` → `0.0`), and the impact score is scaled by `MACRO_SIGNAL_WEIGHT`, which defaults to `0.3` in `AggregationConfig`.
+
+The 0.3 weight means that a macro signal's impact score is reduced to 30% of its raw value before entering the aggregation. This attenuation reflects the inherent uncertainty in macro-to-company impact estimation — a tariff announcement might affect XOM's revenue, but the magnitude depends on exposure profiles, supply chain flexibility, and competitive dynamics that the interpolation engine can only approximate. By weighting macro signals at 0.3 relative to company signals at 1.0, the system ensures that macro intelligence informs the trend without overwhelming direct company-specific evidence.
+
+The recency decay, credibility, and confidence gating for macro signals use the same `compute_signal_weight()` function as company signals. The `published_at` timestamp comes from the global event's source document (the macro news article), and the `source_credibility` and `extraction_confidence` both use the macro impact record's `confidence` field.
+
+### Layer 3 — Competitive Signals (Weight: 0.2)
+
+Competitive signals capture cross-company effects: when a catalyst hits one company, historical patterns suggest how competitors might be affected. They are built by `build_pattern_weighted_signals()` in `services/aggregation/signal_propagation.py` from two sources: `HistoricalPattern` objects (self-company patterns mined by `services/aggregation/pattern_matcher.py`) and `CompetitiveSignalRecord` objects (cross-company propagation signals stored in `competitive_signal_records`).
+
+For historical patterns, the sentiment is derived from the pattern's directional bias (`+1.0` if `bullish_pct > bearish_pct`, `-1.0` otherwise), and the impact score is the pattern's `avg_strength` multiplied by `competitive_signal_weight` (default `0.2` from `CompetitiveConfig`). The `published_at` for recency decay uses the pattern's `data_end` — the most recent data point in the pattern's sample — and the `extraction_confidence` uses the pattern's `pattern_confidence`. Source credibility is set to `1.0` because patterns are derived from validated historical data, and novelty is fixed at `0.5`.
+
+For competitive signal records, the same structure applies: sentiment from `signal_direction`, impact from `signal_strength × competitive_signal_weight`, recency from `computed_at`, and confidence from `pattern_confidence`.
+
+The 0.2 weight makes competitive signals the lightest layer. This is appropriate because competitive signal propagation involves the most inference — the system is predicting how Company B will react based on what happened to Company A in historically similar situations. The signal is valuable as supplementary evidence but should not drive trend direction on its own.
+
+---
+
+## Signal Merging in the Aggregation Engine
+
+The `aggregate_company_window()` function in `services/aggregation/worker.py` orchestrates the merging of all three layers for a single ticker and window. The process follows a clear sequence:
+
+1. **Fetch company impact records** from `document_impact_records` for the ticker within the window's time range.
+2. **Fetch market context** for the ticker from market data tables.
+3. **Build company weighted signals** via `build_weighted_signals()`.
+4. **Check the macro toggle** — query `risk_configs` for the `macro_enabled` flag, then fetch and merge macro signals if enabled.
+5. **Check the competitive toggle** — query `risk_configs` for the `competitive_enabled` flag, then fetch patterns, fetch competitive signals, and merge if enabled.
+6. **Concatenate** all `WeightedSignal` lists into a single list.
+7. **Assemble the `TrendSummary`** from the merged signals.
+
+The concatenation in step 6 is a simple list append — `signals = signals + macro_signals` followed by `signals = signals + pattern_weighted`. There is no re-weighting or normalization at the merge point. The relative influence of each layer is already encoded in the impact scores (scaled by 0.3 for macro, 0.2 for competitive, 1.0 for company) and in the composite weights computed by `compute_signal_weight()`. The `weighted_sentiment_average()` function then naturally produces a sentiment average that reflects these relative weights.
+
+---
+
+## Runtime Toggles and Graceful Degradation
+
+Both the macro and competitive signal layers can be enabled or disabled at runtime through the `risk_configs` PostgreSQL table, without restarting any service. The toggle state is read fresh from the database at the start of every aggregation cycle — there is no caching — so changes take effect on the very next cycle.
+
+The `fetch_macro_enabled()` function in `services/aggregation/worker.py` queries the most recent active `risk_configs` row and reads the `config->>'macro_enabled'` JSON field. If the field is explicitly set to `"true"` or `"false"`, that value overrides the `AggregationConfig` default. If no config row exists or the field is absent, the function returns `None` and the engine falls back to the `AggregationConfig.macro_enabled` default (which is `True`). The `fetch_competitive_enabled()` function follows the identical pattern for the `competitive_enabled` field.
+
+When a layer is disabled, the aggregation engine simply skips the fetch-and-merge step for that layer. Company signals are always computed — they cannot be toggled off. This means the system degrades gracefully: disabling the macro layer produces trends based on company signals alone (plus competitive signals if enabled), and disabling the competitive layer produces trends based on company and macro signals. Disabling both layers reduces the engine to its original single-layer behavior, using only direct document intelligence.
+
+Crucially, disabling a layer does not stop upstream processing. When the macro layer is disabled, the Global Event Classifier continues to classify macro events and the interpolation engine continues to compute `macro_impact_records`. The data accumulates in PostgreSQL. When the layer is re-enabled, the aggregation engine immediately picks up all the macro impact records that were computed while the layer was disabled — there is no data loss or gap in coverage. The same applies to competitive signals: pattern mining and signal propagation continue regardless of the toggle state.
+
+If the competitive signal fetch fails at runtime (for example, due to a database timeout), the aggregation engine catches the exception, logs it, and continues with company and macro signals only. This exception-based graceful degradation ensures that a transient failure in one layer does not block trend computation entirely.
+
+---
+
+## What Comes Next
+
+At this point, every document intelligence record, macro impact record, and competitive signal record has been transformed into a `WeightedSignal` with a composite weight that encodes recency, credibility, novelty, confidence, and market conditions. The three signal layers have been merged into a single list, and the weighted sentiment average has been computed. But a single aggregation cycle produces only a snapshot — a point-in-time view of the evidence. The real power of the system emerges when these snapshots accumulate across multiple documents and time windows, building a case for action. [Page 4 — Trend Aggregation and Accumulating Signals](04-trend-aggregation-and-accumulating-signals.md) explains how the aggregation engine computes `TrendSummary` objects across five time windows, how consecutive same-direction signals strengthen trend confidence and escalate the system's response from neutral observation to actionable trading recommendations, and how contradiction detection and evidence ranking ensure that the trend reflects genuine consensus rather than noise.
diff --git a/docs/intelligence-pipeline-deep-dive/04-trend-aggregation-and-accumulating-signals.md b/docs/intelligence-pipeline-deep-dive/04-trend-aggregation-and-accumulating-signals.md
new file mode 100644
index 0000000..9b20b02
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/04-trend-aggregation-and-accumulating-signals.md
@@ -0,0 +1,267 @@
+# Page 4 — Trend Aggregation and Accumulating Signals
+
+The scoring layer described in [Page 3](03-signal-scoring-and-weighted-signals.md) transforms every intelligence record into a `WeightedSignal` — a document reference paired with a composite weight that encodes recency, credibility, novelty, confidence, and market conditions. Three independent signal layers (Company at weight 1.0, Macro at 0.3, Competitive at 0.2) each produce `WeightedSignal` objects that are concatenated into a single list. But a single list of weighted signals is still just raw material. The aggregation engine in `services/aggregation/worker.py` is where that raw material becomes a decision-grade assessment: a `TrendSummary` object that captures the direction, strength, confidence, contradiction level, and supporting evidence for a ticker across a specific time window. This page explains how that transformation works — from weighted sentiment averages through trend direction derivation, contradiction detection, evidence ranking, and confidence computation — and, critically, how consecutive signals pointing in the same direction accumulate across documents and time windows to escalate the system's response from passive observation to actionable trading recommendations.
+
+For a visual overview of the accumulation and escalation process, see the [Trend Accumulation and Escalation diagram](diagrams/trend-accumulation-escalation.md). For how the three signal layers merge into the aggregation engine, see the [Three-Layer Signal Merging diagram](diagrams/three-layer-signal-merging.md).
+
+---
+
+## Five Time Windows
+
+The aggregation engine does not compute a single trend for each ticker. It computes five, one for each time window defined in `services/aggregation/worker.py`:
+
+| Window | Lookback Duration |
+|--------|-------------------|
+| `intraday` | 12 hours |
+| `1d` | 1 day |
+| `7d` | 7 days |
+| `30d` | 30 days |
+| `90d` | 90 days |
+
+Each window produces an independent `TrendSummary` by fetching all impact records, macro impacts, and competitive signals for the ticker within that window's time range. The `aggregate_company_window()` function in `services/aggregation/worker.py` orchestrates this per-window computation: it determines the time range from the window's lookback duration, fetches `document_impact_records` from PostgreSQL, retrieves market context, builds company weighted signals, checks the macro and competitive runtime toggles (see [Page 3](03-signal-scoring-and-weighted-signals.md) for toggle details), merges any enabled layer signals, and then assembles the `TrendSummary`.
+
+The five-window design serves a specific purpose. Short windows (intraday, 1d) capture fast-moving sentiment shifts — a breaking earnings miss, a sudden regulatory action — while long windows (30d, 90d) reveal sustained trends that persist across many documents and news cycles. A ticker might show a bearish intraday trend after a single negative article, but a neutral 30-day trend because the broader evidence base is balanced. The recommendation engine downstream (described in [Page 5](05-recommendation-generation.md)) evaluates each window's `TrendSummary` independently, so the system can respond to both short-term catalysts and long-term directional shifts.
+
+The `aggregate_company()` function iterates over all effective windows (configurable via `AggregationConfig.windows`, defaulting to all five) and calls `aggregate_company_window()` for each one. This means a single aggregation cycle for one ticker produces up to five `TrendSummary` objects, each reflecting a different temporal perspective on the same underlying evidence.
+
+---
+
+## Trend Direction Derivation
+
+Once the weighted sentiment average has been computed from the merged signal list (see the `weighted_sentiment_average()` function described in [Page 3](03-signal-scoring-and-weighted-signals.md)), the `derive_trend_direction()` function in `services/aggregation/worker.py` maps that numeric value to a `TrendDirection` enum. The rules are evaluated in a specific order, and the first matching rule wins:
+
+1. **Mixed** — If the contradiction score exceeds `0.10` (the `MIXED_THRESHOLD` constant) *and* the absolute value of the average sentiment is below `0.30`, the direction is `MIXED`. This rule fires first because high contradiction with a weak directional signal indicates genuine disagreement in the evidence — the trend is not simply neutral, it is actively contested.
+
+2. **Bullish** — If the average sentiment is `≥ 0.15` (the `BULLISH_THRESHOLD` constant), the direction is `BULLISH`. This means the weight-adjusted evidence leans positive with enough conviction to cross the threshold.
+
+3. **Bearish** — If the average sentiment is `≤ -0.15` (the `BEARISH_THRESHOLD` constant), the direction is `BEARISH`. The symmetric threshold ensures that bullish and bearish classifications require the same magnitude of evidence.
+
+4. **Neutral** — If none of the above conditions are met, the direction is `NEUTRAL`. This covers the range where the average sentiment falls between -0.15 and +0.15 without high contradiction — the evidence is either balanced or insufficient to establish a directional lean.
+
+The mixed-first evaluation order is important. Consider a scenario where five documents are bullish and four are bearish, all with similar weights. The weighted sentiment average might be slightly positive (say, +0.08), which would normally map to neutral. But the contradiction score — computed from the minority/majority weight split — would be high (close to 0.44). The mixed rule catches this case: the evidence is not neutral, it is conflicted. This distinction matters downstream because mixed trends receive different treatment in the recommendation engine than neutral trends.
+
+---
+
+## Contradiction Detection
+
+The contradiction detection module in `services/aggregation/contradiction.py` provides a structured analysis of disagreement within the signal set. Rather than collapsing contradictory evidence into a single number, it produces a `ContradictionResult` containing both an overall score and a list of `DisagreementDetail` objects that explain *where* the disagreement lies.
+
+The `detect_contradictions()` function runs two analyses:
+
+### Sentiment Disagreement
+
+The `_detect_sentiment_disagreement()` function examines whether both positive and negative sentiment signals exist in the signal set. For each signal with a non-zero effective weight (`combined_weight × impact_score > 0`), it classifies the signal as positive or negative based on its `sentiment_value` and accumulates the effective weight for each side. If both sides have at least one signal, it produces a `DisagreementDetail` with dimension `"sentiment"`, listing the document IDs and weights for each side, along with a human-readable description like "Sentiment split: 3 positive vs 2 negative signals (minority weight ratio 38%)".
+
+### Catalyst-Level Disagreement
+
+The `_detect_catalyst_disagreement()` function goes deeper. It groups signals by their `catalyst_type` (earnings, product_launch, regulatory, etc.) using `CatalystEntry` objects built from the `document_impact_records`. Within each catalyst group, it checks whether both positive and negative signals exist. If they do, it produces a `DisagreementDetail` with dimension `"catalyst:<type>"` — for example, `"catalyst:earnings"` when some documents interpret an earnings report positively and others negatively. This catalyst-level analysis is valuable because it pinpoints the specific topic of disagreement rather than just flagging that disagreement exists somewhere in the evidence.
+
+### The Overall Contradiction Score
+
+The `_compute_overall_score()` function computes the backward-compatible scalar contradiction score using the minority/majority weight ratio formula:
+
+```
+contradiction_score = minority_weight / total_weight
+```
+
+where `minority_weight` is the smaller of the positive and negative effective weights, and `total_weight` is their sum. Signals with zero effective weight or neutral sentiment are excluded. The score ranges from `0.0` (complete agreement — all signals point the same direction) to `0.5` (perfect split — positive and negative weights are exactly equal). A score of `0.0` means no contradiction at all. A score above `0.10` combined with a weak average sentiment triggers the mixed direction classification in `derive_trend_direction()`.
+
+The contradiction score also feeds directly into the confidence computation as a penalty, described in the next section. High contradiction reduces the system's confidence in the trend, which in turn affects whether the trend can escalate to actionable recommendations.
+
+---
+
+## Evidence Ranking
+
+Not all documents contributing to a trend are equally important. The `rank_evidence()` function in `services/aggregation/worker.py` delegates to the evidence ranking module (`services/aggregation/evidence.py`) to produce ordered lists of the most influential supporting and opposing documents. The ranking uses a composite scoring approach configured by `EvidenceRankConfig`, considering multiple factors:
+
+- **Weight** — the signal's composite weight from the scoring layer, reflecting recency, credibility, novelty, confidence, and market context.
+- **Impact** — the extraction's impact score for the company, reflecting how significant the document's content is.
+- **Recency** — how recently the document was published, with more recent documents ranked higher.
+- **Confidence** — the extraction confidence, reflecting how reliably the LLM parsed the document.
+
+Signals are split into supporting (positive sentiment) and opposing (negative sentiment) groups. Neutral and mixed sentiment signals are excluded from evidence lists — they do not argue for or against the trend direction. Within each group, signals are sorted by their composite rank score in descending order, and the top entries (up to `MAX_EVIDENCE_REFS = 10` per side) are returned as document ID lists.
+
+The `assemble_trend_with_evidence()` function in `services/aggregation/worker.py` uses the detailed variant `rank_evidence_detailed()` to get `RankedEvidence` objects that include the individual scoring components (weight, impact, recency, confidence, sentiment value). These detailed rankings are persisted to the `trend_evidence` table for auditability, while the document ID lists are stored directly in the `TrendSummary` as `top_supporting_evidence` and `top_opposing_evidence`.
+
+The evidence ranking serves two purposes. First, it provides the recommendation engine with the most relevant documents to cite in its thesis generation (see [Page 5](05-recommendation-generation.md)). Second, it gives human reviewers a quick way to understand *why* the system reached a particular trend assessment — the top-ranked documents are the ones that most influenced the direction and strength.
+
+---
+
+## Confidence Computation
+
+The `compute_trend_confidence()` function in `services/aggregation/worker.py` produces the confidence score for a `TrendSummary`. This score is critical because it directly gates whether a trend can produce actionable recommendations — the eligibility evaluation in `services/recommendation/eligibility.py` requires a minimum confidence of `0.35` to generate any recommendation at all, and higher confidence thresholds control escalation to paper and live trading modes.
+
+Confidence is computed from four components:
+
+### Unique Source Count
+
+The function counts the number of unique document IDs across all active signals (those with `combined_weight > 0`). This count is divided by 15 and capped at `0.8`:
+
+```
+count_factor = min(unique_sources / 15.0, 0.8)
+```
+
+A trend backed by 15 or more unique source documents reaches the maximum count contribution of `0.8`. A trend backed by a single document gets only `0.067`. This component rewards breadth of evidence — a trend confirmed by many independent sources is more trustworthy than one driven by a single article, regardless of how high that article's individual weight might be.
+
+### Average Extraction Credibility
+
+The average credibility weight across all active signals provides a baseline quality measure. If most contributing documents come from high-credibility sources, this component is high. If the evidence is dominated by low-credibility sources, confidence is penalized accordingly.
+
+### Signal Agreement with Sample-Size Dampening
+
+The agreement ratio measures what fraction of directional signals (bullish + bearish, excluding neutral) agree on the majority direction. If 8 out of 10 directional signals are bullish, the raw agreement is `0.8`. But raw agreement is misleading with small sample sizes — 1 out of 1 signals agreeing gives a perfect `1.0` agreement, which is not meaningful.
+
+To address this, the agreement is dampened by a logarithmic sample-size factor:
+
+```
+agreement_dampener = min(1.0, log₂(unique_sources + 1) / log₂(8))
+```
+
+This dampener saturates at `1.0` when `unique_sources` reaches approximately 7 (since `log₂(8) = 3.0` and `log₂(8) = 3.0`). With fewer sources, the dampener reduces the agreement contribution: 1 source gives a dampener of `0.33`, 3 sources give `0.67`, and 7 sources give the full `1.0`. The log₂ scaling means that each additional source provides diminishing marginal improvement to the dampener, which matches the intuition that the jump from 1 to 3 sources is far more meaningful than the jump from 15 to 17.
+
+### Contradiction Penalty
+
+The contradiction score computed by `services/aggregation/contradiction.py` is applied as a direct penalty:
+
+```
+contradiction_penalty = contradiction_score × 0.4
+```
+
+A contradiction score of `0.5` (perfect split) produces a penalty of `0.2`, which is substantial enough to push a moderately confident trend below the eligibility threshold.
+
+### The Combined Formula
+
+The four components are combined as:
+
+```
+confidence = 0.3 × count_factor + 0.3 × avg_credibility + 0.4 × agreement − contradiction_penalty
+```
+
+The result is clamped to `[0.0, 1.0]`. The weighting gives signal agreement the largest share (40%), reflecting the principle that consensus among diverse sources is the strongest indicator of a reliable trend. Source count and credibility each contribute 30%, providing a balanced assessment of evidence breadth and quality. The contradiction penalty can reduce confidence significantly — a highly contradicted trend with a score of 0.4 loses 0.16 points of confidence, which can easily drop it below the 0.35 eligibility gate.
+
+---
+
+## How Accumulating Signals Escalate Decisions
+
+The trend direction, strength, and confidence computed by the aggregation engine are not just descriptive — they directly determine what action the system takes. The escalation path from passive observation to active trading is governed by the eligibility thresholds defined in `services/recommendation/eligibility.py`, and the key insight is that consecutive signals pointing in the same direction naturally strengthen the trend metrics that control this escalation.
+
+### The Escalation Ladder
+
+The `EligibilityConfig` dataclass in `services/recommendation/eligibility.py` defines the thresholds that map trend metrics to actions:
+
+**Neutral (no recommendation).** A trend fails the eligibility gates entirely when confidence is below `0.35`, trend strength is below `0.10`, contradiction exceeds `0.60`, evidence count is below `2`, or the direction is neutral. The `_check_gates()` function evaluates these hard gates — if any gate fails, no recommendation is generated for that window.
+
+**Watch.** A trend that passes the gates but has a direction of mixed, or has strength below `0.25` with confidence below `0.50`, maps to a `WATCH` action via `_determine_action()`. This is the system's way of saying "something is happening, but the evidence is not strong enough to act on." Watch recommendations are always `informational` mode — they are logged for human review but never trigger trades.
+
+**Hold.** When the trend has a clear direction (bullish or bearish) but strength remains below `0.25` while confidence reaches `0.50` or above, the action maps to `HOLD`. This indicates that the directional signal is real but not yet strong enough for a position change. Like watch, hold recommendations are `informational` mode.
+
+**Buy / Sell.** When trend strength reaches `0.25` or above with a bullish direction, the action is `BUY`. With a bearish direction at the same strength threshold, the action is `SELL`. These are the only actions that can escalate beyond informational mode — `_determine_mode()` evaluates whether the recommendation qualifies for `paper_eligible` (confidence ≥ `0.50`) or `live_eligible` (confidence ≥ `0.70`, contradiction ≤ `0.25`, evidence ≥ `5`).
+
+### How Accumulation Drives Escalation
+
+Consider a ticker that starts with no recent intelligence. The first bearish article arrives — a single document with negative sentiment. In the intraday window, this produces:
+
+- **Trend strength** = `|avg_sentiment|` ≈ the absolute weighted sentiment from one signal, likely close to the impact score.
+- **Confidence** = low, because `count_factor = min(1/15, 0.8) = 0.067` and the agreement dampener is only `log₂(2)/log₂(8) = 0.33`.
+- **Direction** = bearish (if the weighted sentiment is ≤ -0.15).
+
+With confidence well below `0.35`, this trend fails the eligibility gate entirely. No recommendation is generated. The system is in the neutral state.
+
+A second bearish article arrives hours later. Now the intraday window has two signals:
+
+- **Unique sources** = 2, so `count_factor = 0.133` and `agreement_dampener = log₂(3)/log₂(8) ≈ 0.53`.
+- **Agreement** = `1.0 × 0.53 = 0.53` (both signals agree on bearish).
+- **Confidence** ≈ `0.3 × 0.133 + 0.3 × avg_cred + 0.4 × 0.53` — likely around `0.35-0.45` depending on credibility.
+
+If confidence crosses `0.35` and strength exceeds `0.10`, the trend passes the eligibility gates. But with strength below `0.25`, the action is `WATCH` or `HOLD` depending on confidence.
+
+A third and fourth bearish article arrive over the next day. The 1-day window now has four agreeing signals:
+
+- **Unique sources** = 4, so `count_factor = 0.267` and `agreement_dampener = log₂(5)/log₂(8) ≈ 0.77`.
+- **Agreement** = `1.0 × 0.77 = 0.77`.
+- **Confidence** ≈ `0.3 × 0.267 + 0.3 × avg_cred + 0.4 × 0.77` — likely `0.50-0.60`.
+- **Strength** = `|avg_sentiment|` — with four bearish signals and no contradicting evidence, this could easily exceed `0.25`.
+
+Now the trend maps to `SELL` with `paper_eligible` mode (confidence ≥ `0.50`). The system has escalated from no recommendation to a paper-eligible sell recommendation purely through the accumulation of consistent bearish evidence.
+
+If the bearish evidence continues — more documents, more sources, higher credibility — confidence climbs further. At confidence ≥ `0.70` with contradiction ≤ `0.25` and evidence ≥ `5`, the recommendation reaches `live_eligible` mode, the highest escalation level.
+
+The same process works in reverse for bullish accumulation: consecutive positive signals strengthen the bullish trend, increase confidence through source diversity and agreement, and escalate from watch through hold to buy.
+
+### The Role of Contradiction in Preventing False Escalation
+
+Accumulation only works when signals agree. If the fifth article about a ticker is bullish while the previous four were bearish, the contradiction score jumps — `minority_weight / total_weight` increases because the minority (bullish) side now has non-zero weight. This has two effects: the contradiction penalty reduces confidence (potentially dropping it below an eligibility threshold), and if the contradiction exceeds `0.10` with `|avg_sentiment| < 0.30`, the direction flips to mixed, which maps to `WATCH` regardless of strength. The system effectively de-escalates when the evidence becomes contested, requiring a clearer consensus before re-escalating.
+
+---
+
+## Trend Projections
+
+After the `TrendSummary` is assembled and persisted, the aggregation engine computes a forward-looking `TrendProjection` via `compute_projection()` in `services/aggregation/projection.py`. Projections estimate where the trend is heading based on current momentum, macro signal decay, and upcoming catalysts. They are advisory — they do not directly trigger recommendations — but they provide valuable context for human reviewers and can inform future automated decision-making.
+
+### Momentum
+
+The `compute_trend_momentum()` function computes the rate of change in signed trend strength between the current and previous aggregation cycles. If the current window shows a bearish trend at strength `0.40` and the previous cycle showed bearish at `0.30`, the momentum is `-0.10` (strengthening bearish). If no previous data is available, the function uses a heuristic: momentum is estimated as half the current signed strength, providing a reasonable baseline for new trends.
+
+Momentum enters the projection as a half-weighted adjustment to the current signed strength:
+
+```
+momentum_projected_signed = direction_sign × current_strength + momentum × 0.5
+```
+
+This means momentum influences the projection but does not dominate it — a strong current trend with weakening momentum still projects as directional, just with reduced strength.
+
+### Macro Decay
+
+The `project_macro_decay()` function estimates how active macro events will evolve over the projection horizon. Each macro event has an `estimated_duration` that maps to a decay half-life:
+
+| Duration | Half-Life |
+|----------|-----------|
+| `short_term` | 1 day |
+| `medium_term` | 7 days |
+| `long_term` | 30 days |
+
+For each event, the function computes the projected remaining impact at the end of the horizon using exponential decay: `future_factor = 2^(−future_age_days / half_life)`. The impact is further scaled by a severity weight (`critical`: 1.0, `high`: 0.75, `moderate`: 0.5, `low`: 0.25). Positive and negative macro impacts are accumulated separately, and the projected macro direction is determined by comparing the two sides — bullish if positive exceeds negative by 20%, bearish if the reverse, mixed if both are present without a clear majority.
+
+When the macro layer is enabled and macro events exist, the projection blends the company-specific momentum projection with the macro trajectory. The macro weight is capped at `0.4` (40% of the blended projection), ensuring that macro signals inform but do not overwhelm the company-specific trend. The blending formula combines the signed company projection with the signed macro projection:
+
+```
+blended = company_weight × momentum_projected + macro_weight × macro_signed
+```
+
+### Driving Factors
+
+The projection records a list of human-readable driving factors that explain what is influencing the projected direction. These include momentum descriptions ("Positive momentum (+0.150) in recent trend strength"), macro impact projections ("Macro signals project bearish impact (strength 0.350) over 7d"), and upcoming catalysts drawn from the trend's `dominant_catalysts` list (limited to the top 3). If no specific factors are identified, a baseline continuation factor is recorded.
+
+### Divergence Detection
+
+After computing the projected direction, the function compares it to the current trend direction. If they differ — for example, the current trend is bearish but the projection is bullish due to decaying negative macro events and positive momentum — the projection is flagged with `diverges_from_current = True` and a divergence driving factor is appended. Divergence signals are particularly valuable because they indicate that the trend may be about to reverse, giving the recommendation engine and human reviewers an early warning.
+
+The projection also flags low confidence when `projected_confidence` falls below the default threshold of `0.3`. Projection confidence starts at 80% of the current trend confidence (reflecting the inherent uncertainty of forward-looking estimates), with a small boost if macro data is available and a further reduction if the macro layer is disabled entirely.
+
+---
+
+## Persistence
+
+Each aggregation cycle persists its results to four PostgreSQL tables, creating a durable record of the trend assessment and its supporting evidence.
+
+### `trend_windows` — Current State
+
+The `persist_trend_summary()` function in `services/aggregation/worker.py` upserts the `TrendSummary` into the `trend_windows` table, keyed by `(entity_type, entity_id, window)`. Each cycle overwrites the previous row for that ticker and window, so `trend_windows` always reflects the most recent assessment. The row includes the trend direction, strength, confidence, contradiction score, disagreement details (as JSON), supporting and opposing evidence document IDs (as JSON arrays), dominant catalysts, material risks, market context, and the generation timestamp.
+
+### `trend_history` — Time-Series Snapshots
+
+Immediately after the upsert, `persist_trend_summary()` also inserts a snapshot row into the `trend_history` table. Unlike `trend_windows`, this table is append-only — every aggregation cycle adds a new row, creating a time-series of how the trend evolved over time. The history table stores the direction, strength, confidence, contradiction score, catalysts, risks, and timestamp. This time-series data powers the trend charts in the dashboard and enables the momentum computation in `services/aggregation/projection.py` by providing the previous cycle's strength and direction. If the history insert fails (for example, if the table does not yet exist in a development environment), the failure is logged at debug level and does not block the main upsert.
+
+### `trend_evidence` — Per-Document Rankings
+
+The `persist_trend_evidence()` function writes detailed evidence ranking rows to the `trend_evidence` table, linked to the `trend_windows` row by its UUID. Each row records a document ID, its role (supporting or opposing), and the individual scoring components: rank score, weight component, impact component, recency component, confidence component, and sentiment value. Non-UUID document IDs (such as synthetic pattern signal IDs like `pattern:AAPL:earnings:7d`) are filtered out before insertion, since the `trend_evidence` table enforces a foreign key to the `documents` table.
+
+### `trend_projections` — Forward-Looking Estimates
+
+The `persist_trend_projection()` function in `services/aggregation/projection.py` inserts the `TrendProjection` into the `trend_projections` table, linked to the `trend_windows` row. The row stores the projected direction, strength, confidence, projection horizon, driving factors (as JSON), macro contribution percentage, divergence flag, and computation timestamp. Like trend history, projections accumulate over time, allowing analysis of how well the system's forward-looking estimates matched subsequent reality.
+
+---
+
+## What Comes Next
+
+At this point, the aggregation engine has transformed weighted signals into `TrendSummary` objects across five time windows, detected contradictions, ranked evidence, computed confidence, and persisted everything to PostgreSQL. The trend metrics — direction, strength, confidence, contradiction score — encode the accumulated weight of evidence for each ticker. But a `TrendSummary` is still an assessment, not an action. The next stage translates these assessments into concrete recommendations: should the system buy, sell, hold, or simply watch? And with what conviction? [Page 5 — Recommendation Generation](05-recommendation-generation.md) explains how the recommendation engine applies data quality suppression, eligibility evaluation, position sizing, thesis generation, and risk classification to convert trend summaries into actionable `Recommendation` objects that the trading engine can execute.
diff --git a/docs/intelligence-pipeline-deep-dive/05-recommendation-generation.md b/docs/intelligence-pipeline-deep-dive/05-recommendation-generation.md
new file mode 100644
index 0000000..dc54cbf
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/05-recommendation-generation.md
@@ -0,0 +1,226 @@
+# Page 5 — Recommendation Generation and Signal-to-Action Translation
+
+The aggregation engine described in [Page 4](04-trend-aggregation-and-accumulating-signals.md) produces `TrendSummary` objects across five time windows for each ticker, encoding the direction, strength, confidence, contradiction level, and supporting evidence accumulated from all three signal layers. But a `TrendSummary` is an assessment — it describes what the evidence says, not what the system should do about it. The recommendation engine is where assessment becomes action. It takes each `TrendSummary`, subjects it to a series of deterministic evaluations, and produces a `Recommendation` object that specifies a concrete action (buy, sell, hold, or watch), an execution mode (informational, paper-eligible, or live-eligible), a position sizing guideline, a human-readable thesis, and a risk classification. Every decision in this pipeline is rule-based and fully traceable — the LLM is only involved in an optional downstream step that rewrites the thesis wording.
+
+The recommendation worker in `services/recommendation/main.py` polls the `stonks:queue:recommendation` Redis queue for jobs, each specifying a ticker and time window. For each job, it delegates to `generate_recommendation()` in `services/recommendation/worker.py`, which orchestrates the full pipeline: fetch the latest trend summary, check for duplicate recommendations, fetch any available trend projection, evaluate data quality suppression, evaluate eligibility, optionally rewrite the thesis via LLM, build the `Recommendation` object, and persist everything to PostgreSQL. For a visual overview of this flow, see the [Recommendation Generation Flow diagram](diagrams/recommendation-generation-flow.md).
+
+---
+
+## Data Quality Suppression
+
+Before the eligibility engine evaluates whether a trend is strong enough to act on, the suppression layer in `services/recommendation/suppression.py` asks a more fundamental question: is the underlying data reliable enough to act on at all? A trend might show high confidence and strong directionality, but if the documents feeding it are stale, poorly extracted, or drawn from a single source type, the apparent signal quality is illusory. The suppression layer acts as a pre-filter on data quality, running before the eligibility engine and forcing any recommendation built on unreliable data to `informational` mode regardless of how strong the trend metrics look.
+
+The `evaluate_suppression()` function accepts a `TrendSummary` and a `DataQualityContext` — a set of metrics about the documents underlying the trend, populated by querying `documents` and `document_intelligence` tables for the evidence document IDs stored in the trend summary. When full document-level metrics are not available (for example, in a development environment without the full document pipeline), the function falls back to `build_quality_context_from_summary()`, which estimates quality metrics from the trend summary's own evidence counts and confidence.
+
+### The Six Data Quality Checks
+
+The suppression evaluation runs six independent checks, each comparing a data quality metric against a configurable threshold defined in `SuppressionConfig`. If any single check fails, the recommendation is suppressed:
+
+1. **Low extraction confidence** — If the average extraction confidence across the evidence documents falls below `0.40` (`min_avg_extraction_confidence`), the underlying LLM extractions are too unreliable. This catches cases where the extractor struggled with document formatting, ambiguous content, or low-quality source material, as described in [Page 2](02-ai-agent-processing-and-extraction.md).
+
+2. **Evidence staleness** — If the most recent evidence document is older than `168` hours (7 days, `max_evidence_staleness_hours`), the trend is based on outdated information. Markets move fast, and a week-old evidence base may no longer reflect current conditions. When documents exist but no timestamp is available, the evidence is conservatively treated as stale.
+
+3. **Low source diversity** — If fewer than `1` distinct source type (`min_source_types`) contributed to the evidence, the signal may be driven by a single unreliable source class. In practice, this check fires when the quality context has documents but all come from the same source type (for example, all news articles with no filings or market data to corroborate).
+
+4. **High extraction failure rate** — If more than `50%` (`max_extraction_failure_rate`) of the documents that should have contributed to the trend failed extraction entirely, the data pipeline is unreliable for this ticker. A high failure rate means the trend summary is built from a biased subset of the available evidence — the failed documents might have told a different story.
+
+5. **Insufficient valid documents** — If fewer than `2` valid (non-failed) documents (`min_valid_documents`) contributed to the trend, there simply is not enough data to act on. A single document, no matter how high-quality, does not provide the corroboration needed for automated trading decisions.
+
+6. **Low data quality score** — The `_compute_data_quality_score()` function computes an overall quality score from three weighted components: extraction confidence (40% weight, normalized against a 0.8 baseline), evidence freshness (30% weight, linear decay over the staleness window), and document coverage (30% weight, combining the valid/total ratio with a count factor that saturates at 10 documents). If this composite score falls below `0.30` (`min_data_quality_score`) and the low-confidence check has not already fired, a general suppression reason is added.
+
+When any check triggers, the `SuppressionResult` records the specific reasons (as `SuppressionReason` enum values) and the computed data quality score. The worker in `services/recommendation/worker.py` uses this result to force the recommendation's mode to `informational` and append a suppression note to the thesis text, ensuring the suppression decision is visible in the audit trail.
+
+### Safety Suppressions: Macro-Only and Pattern-Only Signals
+
+Beyond the six data quality checks, two additional safety suppressions protect against acting on signals that lack company-specific corroboration:
+
+**Macro-only suppression** (`evaluate_macro_only_suppression()`) fires when macro signals are the sole basis for a trend direction — no company-specific signals contributed at all. As described in [Page 3](03-signal-scoring-and-weighted-signals.md), macro signals enter the aggregation engine at a reduced weight of `0.3` relative to company signals. But even at reduced weight, macro signals alone can shift a trend direction if no company-specific evidence exists. When this happens, the recommendation is forced to `informational` mode with a caveat noting that the signal is macro-only and should not be used for automated trading.
+
+**Pattern-only suppression** (`evaluate_pattern_only_suppression()`) applies the same logic to competitive/pattern signals. When pattern-based signals from `services/aggregation/pattern_matcher.py` and `services/aggregation/signal_propagation.py` are the sole contributors — no company-specific or macro signals — the recommendation is suppressed. Historical patterns are valuable context, but acting on them without any current evidence is too speculative for automated trading.
+
+Both safety suppressions are evaluated in the worker after the main suppression check, and both force the mode to `informational` when triggered.
+
+---
+
+## Eligibility Evaluation
+
+Recommendations that survive the suppression layer enter the eligibility evaluation in `services/recommendation/eligibility.py`. This is the core decision logic — a set of deterministic rules that map trend metrics to actions, execution modes, and position sizing. The `evaluate_eligibility()` function is the single entry point, accepting a `TrendSummary` and an `EligibilityConfig` of tunable thresholds.
+
+### Gate Checks
+
+The `_check_gates()` function applies five hard gates. If any gate fails, the trend is ineligible for a recommendation (though the action and mode are still computed for the audit trace):
+
+| Gate | Threshold | Rejection Reason |
+|------|-----------|-----------------|
+| Confidence | ≥ `0.35` | `low_confidence` |
+| Trend strength | ≥ `0.10` | `low_trend_strength` |
+| Contradiction score | ≤ `0.60` | `high_contradiction` |
+| Evidence count | ≥ `2` (supporting + opposing) | `insufficient_evidence` |
+| Direction | ≠ `neutral` | `neutral_direction` |
+
+These gates are intentionally conservative. A confidence threshold of `0.35` means the system needs meaningful evidence breadth and agreement before generating any recommendation at all (see the confidence computation in [Page 4](04-trend-aggregation-and-accumulating-signals.md)). The contradiction ceiling of `0.60` allows moderately contested trends through — only when the evidence is deeply split does the gate reject. The evidence minimum of `2` ensures that no recommendation is ever based on a single document.
+
+When a trend fails any gate, the resulting `EligibilityResult` has `eligible = False` and the mode is forced to `informational`, regardless of what the mode escalation logic would otherwise compute.
+
+### Action Mapping
+
+The `_determine_action()` function maps the trend's direction and strength to one of four action types. The logic evaluates in a specific order:
+
+**Mixed or neutral direction → WATCH.** If the trend direction is `mixed` (high contradiction with weak directional signal) or `neutral`, the action is always `WATCH`. There is no directional conviction to act on.
+
+**Strong directional signal → BUY or SELL.** If the trend strength reaches `0.25` or above (`action_strength_threshold`), the action follows the direction: `BUY` for bullish, `SELL` for bearish. This threshold ensures that only trends with meaningful magnitude trigger position-changing actions.
+
+**Weak directional signal with decent confidence → HOLD.** If the trend has a clear direction (bullish or bearish) but strength remains below `0.25`, the action depends on confidence. If confidence reaches `0.50` or above (`hold_confidence_threshold`), the action is `HOLD` — the system recognizes the directional lean but does not have enough conviction to recommend a position change. Below `0.50` confidence, the action falls to `WATCH`.
+
+This mapping creates the escalation ladder described in [Page 4](04-trend-aggregation-and-accumulating-signals.md): as consecutive signals accumulate and strengthen the trend metrics, the action naturally progresses from WATCH → HOLD → BUY/SELL.
+
+### Mode Escalation
+
+The `_determine_mode()` function determines the highest execution mode allowed for the recommendation. Mode controls whether the recommendation is purely informational, eligible for paper trading, or eligible for live trading:
+
+**WATCH and HOLD → always informational.** These actions do not trigger trades, so they are always `informational` mode. They are logged for human review and dashboard display but never enter the trading engine.
+
+**BUY and SELL → escalation based on signal quality.** For actionable recommendations, mode escalates through three tiers:
+
+- **`informational`** — The default when confidence is below `0.50`. The recommendation is recorded but not eligible for any trading.
+- **`paper_eligible`** — When confidence reaches `0.50` or above (`paper_confidence_threshold`). The recommendation can be picked up by the paper trading engine described in [Page 6](06-trading-decisions-and-execution.md).
+- **`live_eligible`** — The strictest tier, requiring confidence ≥ `0.70` (`live_confidence_threshold`), contradiction ≤ `0.25` (`live_max_contradiction`), and evidence count ≥ `5` (`live_min_evidence`). This triple gate ensures that only high-conviction, well-corroborated, low-contradiction recommendations can trigger live trades.
+
+The evidence count for mode escalation is computed as the sum of supporting and opposing evidence documents, matching the same count used in the gate checks.
+
+---
+
+## Position Sizing
+
+The `_compute_position_sizing()` function in `services/recommendation/eligibility.py` translates signal quality into a portfolio allocation guideline. Position sizing is not a fixed value — it scales dynamically with the confidence and strength of the underlying trend, penalized by contradiction and thin evidence.
+
+### Base and Scaling
+
+The computation starts with a base portfolio allocation of `1%` (`base_portfolio_pct = 0.01`) and scales upward based on two factors:
+
+- **Confidence factor** — `0.8 × confidence` (`confidence_sizing_weight`), reflecting how much the system trusts the trend assessment.
+- **Strength factor** — `0.5 + 0.5 × trend_strength`, ranging from `0.5` (weakest trend) to `1.0` (strongest trend).
+
+The raw portfolio percentage is computed as:
+
+```
+raw_portfolio = base + confidence_factor × strength_factor × (max - base)
+```
+
+where `max` is `10%` (`max_portfolio_pct = 0.10`). At maximum confidence (1.0) and maximum strength (1.0), the raw allocation reaches the full 10%. At typical values (confidence 0.6, strength 0.3), the raw allocation is considerably lower.
+
+### Contradiction Penalty
+
+The contradiction score applies a multiplicative penalty:
+
+```
+portfolio_pct = raw_portfolio × (1.0 − 0.5 × contradiction_score)
+```
+
+A contradiction score of `0.40` reduces the allocation by 20%. A score of `0.0` (no contradiction) applies no penalty. This ensures that contested trends receive smaller position sizes even when they pass the eligibility gates.
+
+### Evidence Count Penalty
+
+Thin evidence further reduces the allocation:
+
+- Fewer than `3` evidence documents → multiply by `0.5` (halved).
+- Fewer than `5` evidence documents → multiply by `0.75`.
+- `5` or more documents → no penalty.
+
+This penalty stacks with the contradiction penalty, so a trend with high contradiction and thin evidence receives a substantially reduced position size.
+
+### Max Loss Scaling
+
+The same scaling logic applies to the maximum loss percentage, which starts at a base of `0.3%` (`base_max_loss_pct = 0.003`) and scales up to `2%` (`max_max_loss_pct = 0.02`). Higher-conviction positions are allowed larger loss tolerances, while low-conviction or contested positions are constrained to tighter stops.
+
+The final `PositionSizing` object (defined in `services/shared/schemas.py`) contains `portfolio_pct` and `max_loss_pct`, both clamped to their respective bounds. This object is embedded in the `Recommendation` and later consumed by the trading engine's own position sizer (described in [Page 6](06-trading-decisions-and-execution.md)), which applies additional portfolio-level constraints.
+
+---
+
+## Thesis Generation
+
+Every recommendation includes a human-readable thesis that explains the reasoning behind the action. Thesis generation happens in two layers: a deterministic assembly that is always present, and an optional LLM rewrite that polishes the wording for trading-eligible recommendations.
+
+### Deterministic Thesis Assembly
+
+The `build_thesis()` function in `services/recommendation/worker.py` constructs a thesis string entirely from the trend data and eligibility result, with no model involvement. The thesis is assembled from several components in order:
+
+1. **Opening** — States the ticker, trend direction, window, strength, and confidence. For example: "AAPL shows a bearish trend over the 7d window with strength 0.35 and confidence 0.62."
+
+2. **Catalysts** — Lists the top three dominant catalysts from the `TrendSummary`, drawn from the evidence ranking described in [Page 4](04-trend-aggregation-and-accumulating-signals.md).
+
+3. **Contradiction note** — If the contradiction score exceeds `0.15`, a note flags the signal disagreement and its magnitude.
+
+4. **Trend projection** — When a `TrendProjection` is available and not flagged as low-confidence, the thesis incorporates the projected direction, strength, and top driving factors. If the projection diverges from the current trend, a divergence note is appended.
+
+5. **Risks** — Lists the top two material risks from the `TrendSummary`.
+
+6. **Evidence count** — States the number of supporting and opposing evidence documents.
+
+7. **Prescriptive action** — States the recommended action and mode (e.g., "Recommendation: SELL (paper eligible).").
+
+The deterministic thesis is always generated and serves as the audit reference. Even when the LLM rewrites the thesis, the deterministic version is preserved in the model metadata for traceability.
+
+### Optional LLM Rewrite via the Thesis-Rewriter Agent
+
+For recommendations that are both eligible and not suppressed, the worker optionally invokes the thesis-rewriter agent to polish the deterministic thesis into analyst-quality prose. The LLM rewrite is implemented in `services/recommendation/thesis_llm.py` and uses the `thesis-rewriter` agent slug, resolved at runtime through the `AgentConfigResolver` in `services/shared/agent_config.py`.
+
+The `AgentConfigResolver` queries the `ai_agents` and `agent_variants` database tables to resolve the active configuration for the `thesis-rewriter` slug, preferring an active variant's model, timeout, and retry settings when one exists. The resolver uses a 60-second TTL in-memory cache to avoid hitting the database on every recommendation. This is the same resolution mechanism used by the document extractor and event classifier agents described in [Page 2](02-ai-agent-processing-and-extraction.md).
+
+The `rewrite_thesis_with_llm()` function builds a prompt from the deterministic thesis and trend context (ticker, window, direction, strength, confidence, contradiction score, catalysts, risks), sends it to the local Ollama instance via HTTP, and returns the rewritten text. The system prompt enforces strict rules: no fabricated information, no numbers or facts not present in the input, under 150 words, neutral professional tone, and only the rewritten thesis text in the response.
+
+The LLM layer is purely additive — if the call fails for any reason (network error, timeout, empty response, token budget exceeded), the original deterministic thesis is returned unchanged. The worker in `services/recommendation/main.py` resolves the thesis-rewriter configuration at startup and refreshes it every 50 jobs to pick up configuration changes without requiring a restart. When no database configuration exists for the `thesis-rewriter` slug, thesis rewriting is silently disabled.
+
+Performance logging for the thesis-rewriter is written to the `agent_performance_log` table, recording success/failure, duration, estimated token counts, and the variant ID. Token budget enforcement checks hourly usage against the variant's configured budget before making the LLM call, preventing runaway costs from high-volume recommendation cycles.
+
+### Risk Classification Prefix
+
+Before the thesis is stored, the `classify_risk()` function in `services/recommendation/worker.py` assigns a risk classification label that is prepended to the thesis text as a `[risk:<level>]` prefix. The classification is computed from a composite score:
+
+| Factor | Contribution |
+|--------|-------------|
+| Contradiction score | `contradiction × 2.0` |
+| Low confidence | `(1.0 − confidence) × 1.5` |
+| Low evidence count | `+1.0` if < 3 docs, `+0.5` if < 5 docs |
+| Rejection reasons | `+0.5` per rejection reason |
+
+The composite score maps to four levels:
+
+| Score Range | Classification |
+|-------------|---------------|
+| ≥ 3.0 | `very_high` |
+| ≥ 2.0 | `high` |
+| ≥ 1.0 | `moderate` |
+| < 1.0 | `low` |
+
+A recommendation with high contradiction (0.4 → contributes 0.8), moderate confidence (0.55 → contributes 0.675), and 4 evidence documents (contributes 0.5) would score 1.975, classifying as `moderate`. The same recommendation with only 2 evidence documents would score 2.475, pushing it to `high`. This classification gives downstream consumers — both the trading engine and human reviewers — a quick risk signal without needing to re-evaluate the underlying metrics.
+
+---
+
+## Persistence
+
+The recommendation pipeline persists its output to three PostgreSQL tables, creating a complete audit trail from trend assessment through decision logic to the final recommendation.
+
+### `recommendations` — The Core Record
+
+The `persist_recommendation()` function in `services/recommendation/worker.py` inserts the `Recommendation` into the `recommendations` table. Each row captures the ticker, action, mode, confidence, time horizon, thesis (including the risk classification prefix and any suppression notes), invalidation conditions (as JSONB), position sizing (portfolio percentage and max loss percentage), model metadata (provider, model name, prompt version, schema version), risk classification, and generation timestamp. The insert returns the recommendation's UUID, which serves as the foreign key for the evidence and risk evaluation tables.
+
+### `recommendation_evidence` — Evidence Citations
+
+For each evidence document referenced in the recommendation, a row is inserted into the `recommendation_evidence` table linking the recommendation UUID to the document UUID, with an evidence type (`supporting` or `opposing`) and a position-based weight that decays with rank: `weight = 1.0 / (1.0 + index × 0.1)`. The first supporting document gets weight `1.0`, the second gets `0.91`, the third `0.83`, and so on. Non-UUID document IDs (such as synthetic pattern signal IDs like `pattern:AAPL:earnings:7d` from the competitive signal layer) are filtered out before insertion, since the table enforces a foreign key to the `documents` table.
+
+### `risk_evaluations` — Decision Audit Trail
+
+The `risk_evaluations` table records the full eligibility decision for each recommendation: whether the trend was eligible, the allowed mode, the list of rejection reasons (as JSONB), and a `risk_checks` JSONB object containing the time horizon, position sizing details, invalidation conditions, and risk classification. This table enables post-hoc analysis of why the system made a particular decision — auditors can trace from the recommendation back through the eligibility evaluation to the underlying trend metrics.
+
+---
+
+## Deduplication
+
+Before running the full evaluation pipeline, the worker checks whether the latest recommendation for the same ticker and time horizon is effectively identical to what would be generated. The `_is_duplicate_recommendation()` function in `services/recommendation/worker.py` compares the previous recommendation's action, mode, and confidence (within a `0.01` tolerance) against the current eligibility result. If all three match, the recommendation is skipped — the underlying trend data has not changed meaningfully since the last cycle. This prevents the system from flooding the `recommendations` table with identical entries on every aggregation cycle, while still generating a new recommendation whenever the trend metrics shift enough to change the action, mode, or confidence.
+
+---
+
+## What Comes Next
+
+At this point, the recommendation engine has translated trend assessments into concrete `Recommendation` objects — each with an action, execution mode, position sizing guideline, thesis, and risk classification — and persisted them alongside their evidence citations and eligibility audit trails. Recommendations marked as `paper_eligible` or `live_eligible` are now available for the trading engine to consume. [Page 6 — Trading Decisions and Execution](06-trading-decisions-and-execution.md) explains how the trading engine polls these recommendations, applies its own pre-trade check sequence (circuit breakers, trading windows, confidence gates, deduplication, declining positions, and max open positions), computes final position sizes with portfolio-level constraints, and submits orders through the broker adapter to Alpaca's paper trading API.
diff --git a/docs/intelligence-pipeline-deep-dive/06-trading-decisions-and-execution.md b/docs/intelligence-pipeline-deep-dive/06-trading-decisions-and-execution.md
new file mode 100644
index 0000000..1ddaf8e
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/06-trading-decisions-and-execution.md
@@ -0,0 +1,199 @@
+# Page 6 — Trading Decisions and Execution
+
+The recommendation engine described in [Page 5](05-recommendation-generation.md) produces `Recommendation` objects with an action, execution mode, position sizing guideline, thesis, and risk classification. Recommendations marked as `paper_eligible` or `live_eligible` are persisted to the `recommendations` table and are now available for the final stage of the pipeline: autonomous trade execution. The trading engine in `services/trading/engine.py` is where intelligence becomes action. It polls eligible recommendations, subjects each one to a strict sequence of pre-trade safety checks, computes a portfolio-aware position size, and — if every gate passes — submits an order through the broker adapter to Alpaca's paper trading API. Every evaluation, whether it results in a trade or a skip, is recorded as a `TradingDecision` in the `trading_decisions` table, creating a complete audit trail from the original document signal through to the broker response.
+
+For a visual overview of the decision flow, see the [Trading Engine Decision Loop diagram](diagrams/trading-engine-decision-loop.md).
+
+---
+
+## The Trading Engine Decision Loop
+
+The `TradingEngine` class in `services/trading/engine.py` is the orchestrator. When `start()` is called, it loads the current portfolio state from PostgreSQL — open positions, reserve pool balance, sector exposure, portfolio heat — and then spawns five concurrent `asyncio` tasks that run for the lifetime of the engine:
+
+1. **`_decision_loop()`** — The core polling loop. Every 60 seconds (configurable via `polling_interval_seconds`), it queries the `recommendations` table for rows where `action IN ('buy', 'sell')`, `mode IN ('paper_eligible', 'live_eligible')`, and `generated_at` is within the last two hours. Recommendations are ordered by confidence descending and capped at 50 per cycle. For each recommendation, the engine fetches the current market price (first from `market_snapshots`, falling back to the Polygon API), then runs the full pre-trade evaluation pipeline described below.
+
+2. **`_stop_loss_monitor()`** — Periodically checks current prices against the stop-loss and take-profit levels maintained by the `StopLossManager` in `services/trading/stop_loss_manager.py`. When a price crosses a stop-loss or take-profit threshold, the monitor submits a sell order to the broker queue. The `StopLossManager` computes initial levels from ATR and risk tier parameters, re-evaluates them when volatility shifts materially (ATR change > 10%), activates trailing stops when the price moves more than 50% toward the take-profit target, and tightens stops proactively when portfolio heat exceeds 80% of the maximum.
+
+3. **`_performance_loop()`** — Computes portfolio-wide performance metrics (total value, unrealized and realized P&L, win rate, Sharpe ratio, drawdown, portfolio heat), persists daily snapshots to `portfolio_snapshots`, checks for daily-loss circuit breaker triggers, evaluates profit-taking opportunities, and synchronizes positions with the database to detect closed positions and trigger reserve pool siphoning.
+
+4. **`_risk_tier_scheduler()`** — Runs once daily at 16:00 ET (market close). It loads the latest `PerformanceMetrics` from `portfolio_snapshots`, computes the reserve pool as a fraction of total portfolio value, and delegates to the `RiskTierController` in `services/trading/risk_tier_controller.py` to determine whether the active risk tier should change. Tier changes are persisted to `risk_tier_history` and take effect immediately for subsequent decision cycles.
+
+5. **`_rebalance_scheduler()`** — Runs weekly on Monday at 09:45 ET (shortly after market open). It loads current positions, evaluates them against the active risk tier's constraints using the `PortfolioRebalancer`, and pushes any rebalance sell orders to `stonks:queue:broker_orders`. The rebalancer respects the circuit breaker — if any breaker is active, the rebalance cycle is skipped entirely.
+
+All five tasks run concurrently within a single `asyncio` event loop. Graceful shutdown via `stop()` cancels all tasks and awaits their completion. If any task encounters an unexpected exception, it logs the error and retries after a brief sleep rather than crashing the engine.
+
+---
+
+## Pre-Trade Check Sequence
+
+When the decision loop picks up a buy recommendation, it calls `evaluate_recommendation()` — a synchronous method that runs the full pre-trade check sequence. The checks are applied in a strict order, and the first failure short-circuits the evaluation with a `skip` decision. This fail-fast design ensures that expensive downstream computations (like position sizing and correlation analysis) are never reached when a simple gate would have rejected the trade.
+
+The six checks, in order:
+
+**a. Circuit breaker check.** The engine calls `self.circuit_breaker.is_active()` on the current `CircuitBreakerState`. If any circuit breaker is active and its cooldown has not expired, the recommendation is skipped with reason `circuit_breaker_active`. The circuit breaker mechanism is described in detail below.
+
+**b. Trading window check.** The `is_within_trading_window()` function verifies that the current time falls within US market hours. Outside the trading window, no orders are submitted — the recommendation is skipped with reason `outside_trading_window`.
+
+**c. Confidence gate.** The recommendation's confidence score is compared against the active risk tier's `min_confidence` threshold. A conservative tier requires confidence ≥ 0.75, moderate requires ≥ 0.55, and aggressive requires ≥ 0.40. If the recommendation's confidence falls below the tier minimum, it is skipped with reason `insufficient_confidence`. This gate ensures that the risk tier's conservatism is enforced before any capital allocation is considered.
+
+**d. Deduplication check.** The engine maintains an in-memory set of processed recommendation IDs (`processed_recommendation_ids`) and also checks Redis via `stonks:dedupe:trading:*` keys (with a 24-hour TTL). If the recommendation has already been evaluated in this engine session or by a previous instance, it is skipped with reason `duplicate_recommendation`. This prevents the same recommendation from generating multiple orders across polling cycles.
+
+**e. Declining positions check.** The `check_declining_positions()` method examines all open positions. If more than 50% of positions have unrealized losses exceeding 2% of their entry value, the engine halts new entries with reason `multiple_declining_positions`. This is a portfolio-level safety valve — when the majority of existing positions are underwater, adding new exposure compounds the risk.
+
+**f. Max open positions check.** The engine enforces a configurable maximum number of concurrent positions (default 10). If the portfolio is already at capacity, the recommendation is skipped with reason `max_positions_reached`.
+
+For sell recommendations, the engine follows a separate, simpler path: it verifies the trading window, looks up the existing position for the ticker, and submits a market sell order for the full position quantity without running the position sizer. Sell decisions still generate a `TradingDecision` audit record and set the Redis deduplication key.
+
+If all six checks pass for a buy recommendation, the engine proceeds to position sizing.
+
+---
+
+## Position Sizing
+
+The `PositionSizer` in `services/trading/position_sizer.py` translates a recommendation's signal quality into a concrete dollar amount and share count, applying a sequential pipeline of adjustments that account for confidence, portfolio composition, sector concentration, correlation, and upcoming earnings events. The sizer operates on the *active pool* — the portion of the portfolio available for trading after subtracting the reserve pool balance.
+
+### Base Sizing
+
+The computation begins with a base allocation percentage derived from the risk tier:
+
+```
+base_allocation_pct = risk_tier.max_position_pct × 0.5
+raw_pct = base_allocation_pct × (confidence / min_confidence)
+```
+
+The base starts at half the tier's maximum position percentage, then scales linearly with how far the recommendation's confidence exceeds the tier minimum. A moderate-tier recommendation with confidence 0.70 against a minimum of 0.55 would produce a raw percentage of `0.05 × (0.70 / 0.55) ≈ 0.0636`, or about 6.4% of the active pool. The raw percentage is clamped to `max_position_pct` (5% for conservative, 10% for moderate, 15% for aggressive) and then converted to a dollar amount against the active pool. An absolute position cap (default $50) provides a hard ceiling regardless of pool size — a safety measure for the paper trading environment.
+
+### Correlation-Aware Diversification
+
+The sizer computes a weighted average correlation between the candidate ticker and all existing positions, using the pairwise correlation matrix that the engine refreshes from 30 days of daily close prices in `market_snapshots`. Each existing position's correlation is weighted by its market value, so larger positions have more influence on the diversification check.
+
+If the weighted average correlation exceeds 0.8, the position is rejected outright — the portfolio already has too much exposure to correlated assets. Between 0.5 and 0.8, the dollar amount is reduced proportionally: a correlation of 0.65 produces a scale factor of `1.0 − (0.65 − 0.5) / (0.8 − 0.5) = 0.5`, halving the position size. Below 0.5, no reduction is applied.
+
+### Sector Exposure Reduction
+
+The sizer checks whether adding the new position would push the sector's total exposure beyond the risk tier's `max_sector_pct` (20% for conservative, 30% for moderate, 40% for aggressive). If the sector is already at its limit, the position is rejected. If the new position would exceed the limit, the dollar amount is reduced to exactly fill the remaining sector capacity.
+
+### Diversification Bonus
+
+When the portfolio holds fewer than three distinct sectors and the candidate ticker belongs to a new sector, the sizer applies a 1.2× bonus to the dollar amount. This incentivizes early diversification — the first few positions are encouraged to spread across sectors rather than concentrating in a single one. The bonus is re-clamped to `max_position_pct` after application to prevent oversized positions.
+
+### Earnings Proximity Adjustment
+
+The sizer checks the earnings calendar for the candidate ticker. If earnings are within one trading day, the position is rejected entirely — the binary risk of an earnings surprise is too high for automated entry. If earnings are within three trading days, the dollar amount is reduced by 50%. Beyond three days, no adjustment is applied.
+
+### Portfolio Heat Check and Share Rounding
+
+After all adjustments, the sizer estimates the new position's contribution to portfolio heat (the aggregate risk from stop-loss distances across all positions). If adding the position would push total heat beyond `max_portfolio_heat × active_pool` (10% for conservative, 20% for moderate, 30% for aggressive), the position is rejected.
+
+Finally, the dollar amount is converted to whole shares via `floor(dollar_amount / current_price)`. If rounding produces zero shares (the position is too small for even one share at the current price), the position is rejected. The final dollar amount is recalculated from the whole-share quantity to reflect the actual capital deployed.
+
+The `PositionSizeResult` returned to the engine includes the dollar amount, share quantity, allocation percentage, a list of human-readable adjustment notes, and a rejected flag with reason if any step failed. These adjustment notes are embedded in the `TradingDecision`'s `decision_trace` for full auditability.
+
+---
+
+## Circuit Breaker
+
+The `CircuitBreaker` in `services/trading/circuit_breaker.py` is a pure computation module that evaluates three independent trigger conditions. It carries no state of its own — the engine manages the `CircuitBreakerState` dataclass and persists trigger events to the `circuit_breaker_events` table and Redis keys under `stonks:trading:circuit_breaker:*`.
+
+### Three Trigger Types
+
+**Daily loss trigger.** When the portfolio's daily P&L loss exceeds 5% of total portfolio value (`daily_loss_pct = 0.05`), the circuit breaker activates. The `check_daily_loss()` method compares the absolute loss ratio against the threshold. The cooldown duration is set to `volatility_pause_hours` (default 2 hours). The performance loop in the engine calls `_check_circuit_breaker_daily_loss()` periodically to evaluate this condition against the latest portfolio metrics. In extreme cases where the drawdown exceeds an emergency threshold, the reserve pool's emergency liquidation mechanism may also be triggered.
+
+**Single position loss trigger.** When any individual position loses more than 15% of its entry value (`single_position_loss_pct = 0.15`), the circuit breaker activates with a ticker-specific cooldown. The `check_single_position()` method evaluates the loss percentage. The cooldown for the affected ticker is set to `ticker_cooldown_hours` (default 48 hours), during which the engine will not re-enter that ticker. The `is_ticker_cooled_down()` method checks whether a specific ticker is still within its cooldown window by consulting the `ticker_cooldowns` dictionary in the `CircuitBreakerState`.
+
+**Volatility trigger (stop-loss clustering).** When three or more stop-losses fire within a 30-minute rolling window (`stop_loss_hits_threshold = 3`, `stop_loss_window_minutes = 30`), the circuit breaker activates. The `check_volatility()` method uses a sliding window algorithm: it sorts the stop-loss timestamps and checks every contiguous subsequence of length `stop_loss_hits_threshold` to see if it fits within the window. This detects rapid-fire stop-loss cascades that indicate extreme market volatility. The cooldown is `volatility_pause_hours` (default 2 hours).
+
+### Cooldown Computation
+
+The `compute_cooldown_expiry()` method calculates when a triggered breaker expires. For `daily_loss` and `volatility` triggers, the expiry is `triggered_at + volatility_pause_hours`. For `single_position` triggers, the expiry is `triggered_at + ticker_cooldown_hours`, giving the affected ticker a longer cooling-off period. The `is_active()` method returns `True` when the breaker is flagged active and the current time has not yet passed the cooldown expiry.
+
+### Redis State Tracking
+
+The engine persists circuit breaker state to Redis under the `stonks:trading:circuit_breaker:*` key pattern (constructed by `trading_cb_key()` in `services/shared/redis_keys.py`). Each trigger type gets its own key — for example, `stonks:trading:circuit_breaker:daily_loss` — storing the activation timestamp and cooldown expiry. This allows the state to survive engine restarts and enables external monitoring tools to query breaker status without accessing the engine's memory.
+
+---
+
+## Reserve Pool
+
+The `ReservePoolController` in `services/trading/reserve_pool.py` manages an untouchable cash reserve that grows from realized trading profits. The reserve serves two purposes: it provides a buffer against drawdowns, and its size relative to the portfolio influences risk tier upgrade decisions.
+
+### Profit Siphoning
+
+When the engine detects a closed position with positive unrealized P&L (via `_sync_positions_and_siphon()` in the performance loop), it calls `siphon_profit()` on the controller. The method transfers a configurable fraction of the realized profit into the reserve — by default 20% (`siphon_pct = 0.20`). Only positive profits are siphoned; losses do not reduce the reserve balance. Each siphon event is recorded in the `reserve_pool_ledger` table with the transfer amount, resulting balance, trigger type (`profit_siphon`), the ticker as reference, and a timestamp.
+
+### High-Water Mark Rebalancing
+
+The `is_high_water()` method returns `True` when the reserve balance exceeds 30% of total portfolio value (`high_water_pct = 0.30`). This signal is consumed by the risk tier scheduler — when the reserve is healthy and other performance criteria are met, the controller may recommend upgrading to a more aggressive tier. The high-water mark acts as a confidence indicator: a large reserve means the system has been consistently profitable and can afford to take on more risk.
+
+### Emergency Liquidation
+
+The `should_emergency_liquidate()` method checks whether the current drawdown exceeds an emergency threshold. When triggered, `emergency_liquidate()` returns the full reserve balance for release back into the active pool. The caller (the engine) is responsible for zeroing the persisted balance and recording the ledger entry. Emergency liquidation is a last resort — it sacrifices the safety buffer to prevent the portfolio from hitting a catastrophic loss level.
+
+### Active Pool Computation
+
+The `compute_active_pool()` method calculates the capital available for trading: `active_pool = total_portfolio_value − reserve_balance`. All position sizing computations use the active pool rather than the total portfolio value, ensuring that the reserve is never inadvertently deployed into new positions.
+
+---
+
+## Risk Tier Auto-Adjustment
+
+The `RiskTierController` in `services/trading/risk_tier_controller.py` evaluates portfolio performance and determines whether the active risk tier should shift. The system supports three tiers — conservative, moderate, and aggressive — each defined by a `RiskTierConfig` dataclass in `services/trading/models.py` with distinct parameter values:
+
+| Parameter | Conservative | Moderate | Aggressive |
+|-----------|-------------|----------|------------|
+| `min_confidence` | 0.75 | 0.55 | 0.40 |
+| `max_position_pct` | 5% | 10% | 15% |
+| `stop_loss_atr_multiplier` | 1.5× | 2.0× | 2.5× |
+| `reward_risk_ratio` | 2.0 | 1.5 | 1.2 |
+| `max_sector_pct` | 20% | 30% | 40% |
+| `max_portfolio_heat` | 10% | 20% | 30% |
+
+The tier controller's `evaluate()` method checks two conditions:
+
+**Downgrade (any one triggers).** If the trailing 30-day win rate drops below 40% or the current drawdown exceeds 15%, the tier steps down by one level (e.g., aggressive → moderate). If the system is already at conservative, no further downgrade is possible.
+
+**Upgrade (all must be true).** If the win rate exceeds 55%, the reserve pool exceeds 20% of total portfolio value, and the current drawdown is below 5%, the tier steps up by one level. The triple requirement ensures that upgrades only happen when the system is performing well, has built a safety cushion, and is not in a drawdown.
+
+The risk tier scheduler in the engine evaluates these conditions daily at market close. When a tier change occurs, it is persisted to the `risk_tier_history` table with the previous tier, new tier, trigger source (`auto_adjustment`), and the metrics that drove the decision (win rate, drawdown, reserve percentage, Sharpe ratio). The new tier takes effect immediately — the engine updates its `_active_risk_tier` reference, and all subsequent decision cycles use the new tier's parameters for confidence gates, position sizing, stop-loss computation, and sector exposure limits.
+
+---
+
+## Order Submission Flow
+
+When `evaluate_recommendation()` returns an `act` decision, the engine constructs an order job and pushes it through a multi-stage submission pipeline that spans two services.
+
+### TradingDecision Persistence
+
+Every evaluation — whether it results in `act` or `skip` — produces a `TradingDecision` dataclass that is persisted to the `trading_decisions` table via `_persist_decision()`. The record captures the recommendation ID, decision outcome, skip reason (if applicable), ticker, computed position size and share quantity, the risk tier at the time of decision, portfolio heat, active pool and reserve pool balances, circuit breaker status, correlation and sector exposure check results, earnings proximity flag, and a `decision_trace` JSONB field containing the full reasoning chain. This creates a complete audit record of every recommendation the engine evaluated and why it acted or declined.
+
+### Order Enqueue
+
+For `act` decisions, the engine builds an order job dictionary containing the trading decision ID, ticker, action (buy or sell), quantity, and order type (market). This job is pushed via `rpush` to the `stonks:queue:broker_orders` Redis queue (constructed by `queue_key(QUEUE_BROKER)` from `services/shared/redis_keys.py`). The engine immediately deducts the estimated order cost from the in-memory active pool to prevent over-allocation across concurrent recommendation evaluations within the same polling cycle.
+
+### Broker Service Processing
+
+The broker service in `services/adapters/broker_service.py` runs as a standalone worker that polls `stonks:queue:broker_orders` via `blpop`. For each order job, `process_order_job()` executes a multi-step pipeline:
+
+1. **Idempotency check.** A deterministic idempotency key is generated from the job's ticker, action, quantity, and trading decision ID. The service checks Redis first (fast path) and then the `orders` table (durable fallback) to prevent duplicate submissions. If a matching key exists, the job is silently dropped.
+
+2. **Risk evaluation.** The service loads the current `PortfolioRiskConfig` from the database and the account's risk state (open positions, daily P&L, sector exposure) from both the database and the Alpaca API. The `evaluate_order()` function runs the proposed order through a set of risk checks — position limits, sector concentration, daily loss thresholds — and produces an evaluation result. The evaluation is persisted to the `risk_evaluations` table regardless of outcome.
+
+3. **Alpaca submission.** If the risk evaluation passes, the service calls `submit_order()` on the `AlpacaBrokerAdapter` in `services/adapters/broker_adapter.py`. The adapter constructs the Alpaca REST API payload (symbol, quantity, side, order type, time in force) and submits it to `paper-api.alpaca.markets/v2/orders` with an idempotency key header. The adapter follows a fail-closed policy: any network error or ambiguous response returns a rejected `OrderResponse` rather than risking duplicate orders.
+
+4. **Persistence and audit trail.** The `persist_order()` function writes the order to the `orders` table with the full request and response details, risk evaluation results, and the recommendation ID for traceability. When the order is filled, the fill details (price, quantity) are recorded. Order events are published to the analytical lakehouse via MinIO for downstream analysis. The Redis idempotency marker is set after successful persistence to prevent reprocessing.
+
+The result is a complete chain of custody: from the original document that produced a signal (Pages [1](01-data-ingestion-and-preparation.md)–[2](02-ai-agent-processing-and-extraction.md)), through signal scoring ([Page 3](03-signal-scoring-and-weighted-signals.md)) and trend aggregation ([Page 4](04-trend-aggregation-and-accumulating-signals.md)), to the recommendation ([Page 5](05-recommendation-generation.md)), the trading decision, the risk evaluation, and the broker response — every step is persisted and linked by foreign keys. The `trading_decisions` table links to `recommendations` via `recommendation_id`, the `orders` table links back to both, and the `positions` and `portfolio_snapshots` tables capture the portfolio impact over time.
+
+For additional reference on the trading engine's configuration, queue topology, and database tables, see [docs/services.md](../services.md).
+
+---
+
+## Conclusion: From Raw Data to Trade Execution
+
+This six-page series has traced the full intelligence-to-decision pipeline in Stonks Oracle, from the moment raw data enters the system to the moment an order reaches the broker.
+
+It began with [Page 1](01-data-ingestion-and-preparation.md), where the scheduler orchestrates ingestion cycles across four data sources — Polygon news, SEC EDGAR filings, Polygon market data, and macro news APIs — and the parser normalizes raw content into structured documents ready for AI processing. [Page 2](02-ai-agent-processing-and-extraction.md) described how the Document Intelligence Extractor and Global Event Classifier agents use LLM inference to produce structured JSON intelligence, with hot-swappable model configurations and a robust JSON repair pipeline. [Page 3](03-signal-scoring-and-weighted-signals.md) explained how raw extraction output is transformed into `WeightedSignal` objects through a composite formula that balances recency, credibility, novelty, and market context across three independent signal layers. [Page 4](04-trend-aggregation-and-accumulating-signals.md) showed how the aggregation engine merges these signals across five time windows, detecting contradictions, ranking evidence, and computing trend projections — with consecutive same-direction signals accumulating to escalate the system's response from neutral through watch and hold to buy or sell. [Page 5](05-recommendation-generation.md) covered the translation of trend assessments into actionable recommendations through data quality suppression, eligibility evaluation, position sizing, thesis generation, and risk classification.
+
+And here in Page 6, the pipeline reached its terminus: the trading engine's decision loop polling those recommendations, subjecting each to circuit breaker checks, confidence gates, deduplication, portfolio health assessments, and a multi-step position sizer — then submitting approved orders through the broker adapter to Alpaca's paper trading API, with every decision recorded in a fully auditable trail from signal to execution.
+
+The pipeline is designed to be conservative by default and transparent throughout. Every stage applies its own safety checks — deduplication at ingestion, confidence gates at extraction, contradiction detection at aggregation, suppression at recommendation, and circuit breakers at trading. The system can be tuned through runtime configuration (risk tier parameters, suppression thresholds, signal layer toggles in `risk_configs`) without code changes or restarts. And the complete audit trail — from `documents` through `document_intelligence`, `document_impact_records`, `trend_windows`, `recommendations`, `trading_decisions`, and `orders` — means that any trade can be traced back to the specific documents, signals, and decisions that produced it.
diff --git a/docs/intelligence-pipeline-deep-dive/diagrams/.gitkeep b/docs/intelligence-pipeline-deep-dive/diagrams/.gitkeep
new file mode 100644
index 0000000..8b13789
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/diagrams/.gitkeep
@@ -0,0 +1 @@
+
diff --git a/docs/intelligence-pipeline-deep-dive/diagrams/ingestion-to-extraction-flow.md b/docs/intelligence-pipeline-deep-dive/diagrams/ingestion-to-extraction-flow.md
new file mode 100644
index 0000000..21daf28
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/diagrams/ingestion-to-extraction-flow.md
@@ -0,0 +1,81 @@
+# Ingestion-to-Extraction Flow
+
+```mermaid
+flowchart TD
+ subgraph Scheduler["Scheduler\nservices/scheduler/app.py"]
+ S1["schedule_cycle()"]
+ S2["Cadence check\nmarket_api: 300s\nnews_api: 300s\nfilings_api: 3600s\nmacro_news: 600s"]
+ S3["Rate limit check\ncheck_rate_limit()"]
+ S1 --> S2 --> S3
+ end
+
+ S3 -->|"rpush"| Q_ING["stonks:queue:ingestion"]
+
+ Q_ING -->|"lpop"| ING
+
+ subgraph ING["Ingestion Worker\nservices/ingestion/worker.py"]
+ direction TB
+ AD["Adapter Dispatch\nprocess_job()"]
+ AD --> PA["PolygonMarketAdapter\nservices/adapters/market_adapter.py"]
+ AD --> PB["PolygonNewsAdapter\nservices/adapters/news_adapter.py"]
+ AD --> PC["SECEdgarAdapter\nservices/adapters/filings_adapter.py"]
+ AD --> PD["MacroNewsAdapter\nservices/adapters/macro_news_adapter.py"]
+ AD --> PE["WebScrapeAdapter\nservices/adapters/web_scrape_adapter.py"]
+ end
+
+ ING -->|"Content hash check\nstonks:dedupe:*\nTTL 24h"| REDIS_DEDUPE[("Redis\nDedupe Markers")]
+
+ ING -->|"upload_raw_artifact()"| MINIO_RAW
+
+ subgraph MINIO_RAW["MinIO Raw Storage"]
+ B1["stonks-raw-market"]
+ B2["stonks-raw-news"]
+ B3["stonks-raw-filings"]
+ end
+
+ ING -->|"persist_ingestion_items()"| PG_ING
+
+ subgraph PG_ING["PostgreSQL"]
+ T1["documents"]
+ T2["ingestion_runs"]
+ T3["document_company_mentions"]
+ end
+
+ ING -->|"rpush new doc IDs"| Q_PARSE["stonks:queue:parsing"]
+
+ Q_PARSE -->|"lpop"| PARSER
+
+ subgraph PARSER["Parser Worker\nservices/parser/worker.py"]
+ P1["fetch_html() → parse_html()"]
+ P2["Quality scoring\nconfidence: high / medium / low"]
+ P3["Company mention detection\ndetect_company_mentions()"]
+ P4["Routing decision"]
+ P1 --> P2 --> P3 --> P4
+ end
+
+ PARSER -->|"upload_normalized_text()\nupload_parser_output()"| MINIO_NORM["MinIO\nstonks-normalized"]
+ PARSER -->|"update_document_parse_results()"| PG_ING
+
+ P4 -->|"doc_type = macro_event"| Q_MACRO["stonks:queue:macro_classification"]
+ P4 -->|"doc_type ≠ macro_event"| Q_EXT["stonks:queue:extraction"]
+
+ Q_EXT -->|"lpop"| EXT
+ Q_MACRO -->|"lpop"| EXT
+
+ subgraph EXT["Extractor Worker\nservices/extractor/main.py"]
+ E1["Document Intelligence\nExtractor agent\nslug: document-extractor"]
+ E2["Global Event Classifier\nslug: event-classifier\nservices/extractor/event_classifier.py"]
+ E3["persist_extraction()\nservices/extractor/worker.py"]
+ end
+
+ EXT -->|"persist to"| PG_EXT
+
+ subgraph PG_EXT["PostgreSQL"]
+ T4["document_intelligence"]
+ T5["document_impact_records"]
+ T6["global_events"]
+ T7["macro_impact_records"]
+ end
+
+ EXT -->|"rpush"| Q_AGG["stonks:queue:aggregation"]
+```
diff --git a/docs/intelligence-pipeline-deep-dive/diagrams/recommendation-generation-flow.md b/docs/intelligence-pipeline-deep-dive/diagrams/recommendation-generation-flow.md
new file mode 100644
index 0000000..4fe510e
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/diagrams/recommendation-generation-flow.md
@@ -0,0 +1,80 @@
+# Recommendation Generation Flow
+
+```mermaid
+flowchart TD
+ Q_REC["stonks:queue:recommendation"] -->|"lpop"| WORKER["Recommendation Worker\nservices/recommendation/main.py"]
+
+ WORKER --> FETCH["Fetch TrendSummary\nfrom trend_windows\nfor ticker + window"]
+
+ FETCH --> SUPP
+
+ subgraph SUPP["Data Quality Suppression\nservices/recommendation/suppression.py"]
+ S1["extraction confidence < 0.40?"]
+ S2["evidence staleness > 168h?"]
+ S3["source diversity < 1 type?"]
+ S4["extraction failure rate > 50%?"]
+ S5["valid documents < 2?"]
+ S6["data quality score < 0.30?"]
+ S7["Macro-only signal?\nevaluate_macro_only_suppression()"]
+ S8["Pattern-only signal?\nevaluate_pattern_only_suppression()"]
+ end
+
+ SUPP -->|"Any check fails:\nsuppressed = true\nmode → informational"| ELIG
+ SUPP -->|"All checks pass"| ELIG
+
+ subgraph ELIG["Eligibility Evaluation\nservices/recommendation/eligibility.py"]
+ direction TB
+ G["Gate Checks"]
+ G1["confidence ≥ 0.35"]
+ G2["strength ≥ 0.10"]
+ G3["contradiction ≤ 0.60"]
+ G4["evidence ≥ 2"]
+ G5["direction ≠ neutral"]
+ G --> G1 & G2 & G3 & G4 & G5
+
+ G1 & G2 & G3 & G4 & G5 --> ACT["Action Mapping"]
+ ACT --> A1["BUY: bullish + strength ≥ 0.25"]
+ ACT --> A2["SELL: bearish + strength ≥ 0.25"]
+ ACT --> A3["HOLD: directional + confidence ≥ 0.50"]
+ ACT --> A4["WATCH: otherwise"]
+
+ A1 & A2 & A3 & A4 --> MODE["Mode Escalation"]
+ MODE --> M1["informational\n(default for HOLD/WATCH)"]
+ MODE --> M2["paper_eligible\nconfidence ≥ 0.50"]
+ MODE --> M3["live_eligible\nconfidence ≥ 0.70\ncontradiction ≤ 0.25\nevidence ≥ 5"]
+ end
+
+ ELIG --> SIZING
+
+ subgraph SIZING["Position Sizing\nservices/recommendation/eligibility.py"]
+ PS1["base = 1% portfolio"]
+ PS2["scale by confidence × strength\nup to 10% max"]
+ PS3["contradiction penalty\n−0.5 × contradiction_score"]
+ PS4["evidence count penalty\n< 3 docs → ×0.5\n< 5 docs → ×0.75"]
+ end
+
+ SIZING --> THESIS
+
+ subgraph THESIS["Thesis Generation"]
+ TH1["Deterministic thesis\nassembled from trend data"]
+ TH2["Optional LLM rewrite\nthesis-rewriter agent\nservices/recommendation/thesis_llm.py"]
+ TH1 --> TH2
+ end
+
+ THESIS --> RISK
+
+ subgraph RISK["Risk Classification"]
+ RC1["low"]
+ RC2["moderate"]
+ RC3["high"]
+ RC4["very_high"]
+ end
+
+ RISK --> PERSIST
+
+ subgraph PERSIST["Persistence — PostgreSQL"]
+ P1["recommendations"]
+ P2["recommendation_evidence"]
+ P3["risk_evaluations"]
+ end
+```
diff --git a/docs/intelligence-pipeline-deep-dive/diagrams/three-layer-signal-merging.md b/docs/intelligence-pipeline-deep-dive/diagrams/three-layer-signal-merging.md
new file mode 100644
index 0000000..0eff51d
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/diagrams/three-layer-signal-merging.md
@@ -0,0 +1,52 @@
+# Three-Layer Signal Merging
+
+```mermaid
+flowchart TD
+ subgraph Layer1["Layer 1 — Company Signals"]
+ DIR["document_impact_records\n(per-company extraction output)"]
+ DIR -->|"build_weighted_signals()"| WS1["WeightedSignal[]\nweight = 1.0 (full)"]
+ end
+
+ subgraph Layer2["Layer 2 — Macro Signals"]
+ MIR["macro_impact_records\n(global event interpolation)"]
+ MIR -->|"build_macro_weighted_signals()"| WS2["WeightedSignal[]\nimpact × MACRO_SIGNAL_WEIGHT\n(0.3)"]
+ TOGGLE_M{"macro_enabled\nin risk_configs?"}
+ TOGGLE_M -->|"true"| MIR
+ TOGGLE_M -->|"false"| SKIP_M["Layer skipped\ngraceful degradation"]
+ end
+
+ subgraph Layer3["Layer 3 — Competitive Signals"]
+ CSR["competitive_signal_records\n(pattern mining + propagation)"]
+ CSR -->|"build_pattern_weighted_signals()\nservices/aggregation/signal_propagation.py"| WS3["WeightedSignal[]\nimpact × COMPETITIVE_SIGNAL_WEIGHT\n(0.2)"]
+ TOGGLE_C{"competitive_enabled\nin risk_configs?"}
+ TOGGLE_C -->|"true"| CSR
+ TOGGLE_C -->|"false"| SKIP_C["Layer skipped\ngraceful degradation"]
+ end
+
+ WS1 --> MERGE["Concatenate all WeightedSignal lists"]
+ WS2 --> MERGE
+ WS3 --> MERGE
+
+ MERGE --> AGG
+
+ subgraph AGG["Aggregation Engine\nservices/aggregation/worker.py"]
+ A1["weighted_sentiment_average()"]
+ A2["detect_contradictions()\nservices/aggregation/contradiction.py"]
+ A3["derive_trend_direction()"]
+ A4["compute_trend_confidence()"]
+ A5["rank_evidence()"]
+ A1 --> A2 --> A3 --> A4 --> A5
+ end
+
+ AGG -->|"assemble_trend_summary()"| TS["TrendSummary\nservices/shared/schemas.py"]
+
+ TS -->|"persist_trend_summary()"| PG_TREND
+
+ subgraph PG_TREND["PostgreSQL"]
+ TW["trend_windows\n(upserted each cycle)"]
+ TH["trend_history\n(time-series snapshots)"]
+ TE["trend_evidence\n(per-document rankings)"]
+ end
+
+ AGG -->|"rpush"| Q_REC["stonks:queue:recommendation"]
+```
diff --git a/docs/intelligence-pipeline-deep-dive/diagrams/trading-engine-decision-loop.md b/docs/intelligence-pipeline-deep-dive/diagrams/trading-engine-decision-loop.md
new file mode 100644
index 0000000..eaf2a40
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/diagrams/trading-engine-decision-loop.md
@@ -0,0 +1,94 @@
+# Trading Engine Decision Loop
+
+```mermaid
+flowchart TD
+ subgraph ENGINE["Trading Engine\nservices/trading/engine.py"]
+ direction TB
+ TASKS["5 Concurrent Async Tasks"]
+ T1["_decision_loop()\n60s polling interval"]
+ T2["_stop_loss_monitor()"]
+ T3["_performance_loop()"]
+ T4["_risk_tier_scheduler()"]
+ T5["_rebalance_scheduler()"]
+ TASKS --> T1 & T2 & T3 & T4 & T5
+ end
+
+ T1 --> POLL["Poll recommendations table\naction IN (buy, sell)\nmode IN (paper_eligible, live_eligible)\ngenerated_at > NOW() − 2h"]
+
+ POLL --> EVAL["evaluate_recommendation()"]
+
+ EVAL --> CHK_A
+
+ subgraph PRETRADE["Pre-Trade Check Sequence\n(first failure short-circuits)"]
+ direction TB
+ CHK_A["a. Circuit Breaker active?\nservices/trading/circuit_breaker.py\nTriggers: daily_loss, single_position, volatility"]
+ CHK_B["b. Trading Window?\nis_within_trading_window()"]
+ CHK_C["c. Confidence Gate\nconfidence ≥ risk_tier.min_confidence"]
+ CHK_D["d. Deduplication\nRec ID in processed set?\nRedis: stonks:dedupe:trading:*"]
+ CHK_E["e. Declining Positions\n> 50% positions down > 2%"]
+ CHK_F["f. Max Open Positions\nopen_count ≥ max (default 10)"]
+
+ CHK_A -->|"pass"| CHK_B
+ CHK_B -->|"pass"| CHK_C
+ CHK_C -->|"pass"| CHK_D
+ CHK_D -->|"pass"| CHK_E
+ CHK_E -->|"pass"| CHK_F
+ end
+
+ CHK_A & CHK_B & CHK_C & CHK_D & CHK_E & CHK_F -->|"fail"| SKIP["TradingDecision\ndecision = skip\n+ skip_reason"]
+
+ CHK_F -->|"pass"| SIZER
+
+ subgraph SIZER["Position Sizing\nservices/trading/position_sizer.py"]
+ direction TB
+ SZ1["Base sizing\nrisk_tier.max_position_pct × 0.5\n× (confidence / min_confidence)"]
+ SZ2["Correlation reduction\nweighted avg corr > 0.8 → reject\n> 0.5 → proportional reduction"]
+ SZ3["Sector exposure\ncap at risk_tier.max_sector_pct"]
+ SZ4["Diversification bonus\n1.2× for new sector (< 3 sectors)"]
+ SZ5["Earnings proximity\n≤ 1 day → reject\n≤ 3 days → 50% reduction"]
+ SZ6["Absolute position cap"]
+ SZ7["Portfolio heat check\nmax_portfolio_heat × active_pool"]
+ SZ8["Share rounding\nfloor(dollar / price)"]
+
+ SZ1 --> SZ2 --> SZ3 --> SZ4 --> SZ5 --> SZ6 --> SZ7 --> SZ8
+ end
+
+ SIZER -->|"rejected"| SKIP
+ SIZER -->|"approved"| ACT["TradingDecision\ndecision = act\nshares, dollar amount"]
+
+ ACT --> PERSIST_TD["Persist to\ntrading_decisions"]
+
+ ACT --> ORDER["Build order job\n{ticker, action, side,\nquantity, order_type}"]
+
+ ORDER -->|"rpush"| Q_BROKER["stonks:queue:broker_orders"]
+
+ Q_BROKER --> BROKER["Broker Adapter\nAlpaca paper trading\nservices/adapters/broker_adapter.py"]
+
+ BROKER --> AUDIT
+
+ subgraph AUDIT["Audit Trail — PostgreSQL"]
+ AU1["orders"]
+ AU2["positions"]
+ AU3["portfolio_snapshots"]
+ end
+
+ subgraph CB_DETAIL["Circuit Breaker Detail\nservices/trading/circuit_breaker.py"]
+ CB1["daily_loss\nportfolio loss > 5%\ncooldown: volatility_pause_hours"]
+ CB2["single_position\nposition loss > 15%\ncooldown: ticker_cooldown_hours (48h)"]
+ CB3["volatility\n≥ 3 stop-losses in 30min\ncooldown: volatility_pause_hours (2h)"]
+ CB4["Redis state\nstonks:trading:circuit_breaker:*"]
+ end
+
+ subgraph RESERVE["Reserve Pool\nservices/trading/reserve_pool.py"]
+ RP1["Profit siphoning: 20%"]
+ RP2["High-water rebalance: 30%"]
+ RP3["Emergency liquidation"]
+ RP4["reserve_pool_ledger"]
+ end
+
+ subgraph RISK_TIER["Risk Tier Auto-Adjustment\nservices/trading/risk_tier_controller.py"]
+ RT1["Evaluate: Sharpe ratio,\ndrawdown, win rate"]
+ RT2["conservative → moderate → aggressive"]
+ RT3["risk_tier_history"]
+ end
+```
diff --git a/docs/intelligence-pipeline-deep-dive/diagrams/trend-accumulation-escalation.md b/docs/intelligence-pipeline-deep-dive/diagrams/trend-accumulation-escalation.md
new file mode 100644
index 0000000..3514776
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/diagrams/trend-accumulation-escalation.md
@@ -0,0 +1,62 @@
+# Trend Accumulation and Escalation
+
+```mermaid
+flowchart TD
+ subgraph Windows["Five Time Windows\nservices/aggregation/worker.py"]
+ W1["intraday (12h)"]
+ W2["1d (1 day)"]
+ W3["7d (7 days)"]
+ W4["30d (30 days)"]
+ W5["90d (90 days)"]
+ end
+
+ W1 & W2 & W3 & W4 & W5 --> SIGNALS
+
+ SIGNALS["Fetch signals per window\nCompany + Macro + Competitive\n→ WeightedSignal[]"]
+
+ SIGNALS --> SENT["weighted_sentiment_average()\nCompute avg sentiment across signals"]
+
+ SENT --> DIR
+
+ subgraph DIR["derive_trend_direction()"]
+ D1["avg_sentiment ≥ 0.15 → BULLISH"]
+ D2["avg_sentiment ≤ −0.15 → BEARISH"]
+ D3["contradiction > 0.10\nAND |avg| < 0.30 → MIXED"]
+ D4["otherwise → NEUTRAL"]
+ end
+
+ DIR --> CONF
+
+ subgraph CONF["compute_trend_confidence()"]
+ C1["Unique source count\ncaps at 15 → 0.8 contribution"]
+ C2["Avg extraction credibility"]
+ C3["Signal agreement ratio\ndampened by log₂(n+1)/log₂(8)\nsaturates ~7 unique sources"]
+ C4["Contradiction penalty\n−0.4 × contradiction_score"]
+ C5["confidence = 0.3×count + 0.3×credibility\n+ 0.4×agreement − penalty"]
+ end
+
+ CONF --> STRENGTH["trend_strength = |avg_sentiment|\nclamped to [0, 1]"]
+
+ STRENGTH --> ESC
+
+ subgraph ESC["Escalation Path\n(via eligibility thresholds)"]
+ direction TB
+ E1["NEUTRAL\nconfidence < 0.35\nOR strength < 0.10\nOR direction = neutral"]
+ E2["WATCH\nstrength < 0.25\nAND confidence < 0.50"]
+ E3["HOLD\nstrength < 0.25\nAND confidence ≥ 0.50"]
+ E4["BUY / SELL\nstrength ≥ 0.25\nAND direction = bullish/bearish"]
+
+ E1 -->|"More signals\nsame direction"| E2
+ E2 -->|"Confidence grows\nmore unique sources"| E3
+ E3 -->|"Strength exceeds 0.25\naccumulated evidence"| E4
+ end
+
+ ESC --> PERSIST
+
+ subgraph PERSIST["Persistence"]
+ P1["trend_windows\n(upserted each cycle)"]
+ P2["trend_history\n(time-series snapshots)"]
+ P3["trend_evidence\n(per-document rankings)"]
+ P4["trend_projections\nservices/aggregation/projection.py"]
+ end
+```
diff --git a/docs/intelligence-pipeline-deep-dive/diagrams/weighted-signal-computation.md b/docs/intelligence-pipeline-deep-dive/diagrams/weighted-signal-computation.md
new file mode 100644
index 0000000..bd0b019
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/diagrams/weighted-signal-computation.md
@@ -0,0 +1,58 @@
+# Weighted Signal Computation
+
+```mermaid
+flowchart TD
+ DOC["Document Signal Input\n(published_at, source_credibility,\nnovelty_score, extraction_confidence,\nmarket_ctx)"]
+
+ DOC --> GATE
+ DOC --> REC
+ DOC --> CRED
+ DOC --> NOV
+ DOC --> MKT
+
+ subgraph GATE["Confidence Gate"]
+ G1["extraction_confidence ≥ 0.2?"]
+ G1 -->|"Yes"| G2["gate = 1.0"]
+ G1 -->|"No"| G3["gate = 0.0\n(signal zeroed out)"]
+ end
+
+ subgraph REC["Recency Decay"]
+ R1["w = 2^(−age_hours / half_life)"]
+ R2["Half-lives per window:\nintraday: 2h\n1d: 12h\n7d: 72h\n30d: 240h\n90d: 720h"]
+ R3["Floor: min_recency_weight = 0.01"]
+ R1 --- R2
+ R1 --- R3
+ end
+
+ subgraph CRED["Source Credibility"]
+ C1["Clamp to [0.1, 1.0]"]
+ C2["Apply exponent\n(default 1.0)"]
+ C1 --> C2
+ end
+
+ subgraph NOV["Novelty Bonus"]
+ N1["bonus = novelty_score × 0.25"]
+ N2["Range: [0.0, 0.25]\n(up to 25% boost)"]
+ N1 --- N2
+ end
+
+ subgraph MKT["Market Context Multiplier"]
+ M1["Volatility boost\nlog₁₊(excess) × 0.15\ncapped at 0.30"]
+ M2["Volume surge boost\nvolume_change > 50% → +0.15"]
+ M3["multiplier = 1.0 + boost\n(always ≥ 1.0)"]
+ M1 --> M3
+ M2 --> M3
+ end
+
+ GATE --> FORMULA
+ REC --> FORMULA
+ CRED --> FORMULA
+ NOV --> FORMULA
+ MKT --> FORMULA
+
+ FORMULA["combined = gate × recency × credibility\n× (1 + novelty_bonus)\n× market_context_multiplier"]
+
+ FORMULA --> SW["SignalWeight\nservices/aggregation/scoring.py"]
+
+ SW --> WS["WeightedSignal\n{ document_id, weight: SignalWeight,\nsentiment_value, impact_score }"]
+```
diff --git a/docs/intelligence-pipeline-deep-dive/index.md b/docs/intelligence-pipeline-deep-dive/index.md
new file mode 100644
index 0000000..2ce9565
--- /dev/null
+++ b/docs/intelligence-pipeline-deep-dive/index.md
@@ -0,0 +1,40 @@
+# Intelligence Pipeline Deep Dive
+
+This document series provides a narrative walkthrough of the full intelligence-to-decision pipeline in Stonks Oracle. Unlike the existing service reference and API documentation, these pages tell the story of how raw data enters the system, gets processed by AI agents, produces structured signals, accumulates into trend summaries, and ultimately drives autonomous trading decisions.
+
+Each page covers one stage of the pipeline and ends with a transition to the next, so you can read the series end-to-end or jump directly to the stage you need. Diagrams are stored as standalone Mermaid files that can be rendered independently or embedded in other documents.
+
+---
+
+## Table of Contents
+
+1. [Data Ingestion and Preparation](01-data-ingestion-and-preparation.md) — How raw data from Polygon.io, SEC EDGAR, and macro news APIs enters the system, gets deduplicated, stored, parsed, and routed for AI processing.
+2. [AI Agent Processing and Structured Extraction](02-ai-agent-processing-and-extraction.md) — How the Document Intelligence Extractor and Global Event Classifier agents use LLM inference to produce structured JSON intelligence from documents.
+3. [Signal Scoring and the WeightedSignal Abstraction](03-signal-scoring-and-weighted-signals.md) — How raw extraction output is transformed into weighted signals through confidence gating, recency decay, source credibility, novelty bonuses, and market context multipliers.
+4. [Trend Aggregation and Accumulating Signals](04-trend-aggregation-and-accumulating-signals.md) — How the aggregation engine merges weighted signals across five time windows, detects contradictions, ranks evidence, and escalates trend strength as consecutive signals accumulate.
+5. [Recommendation Generation](05-recommendation-generation.md) — How trend summaries pass through data quality suppression, eligibility evaluation, position sizing, thesis generation, and risk classification to produce actionable recommendations.
+6. [Trading Decisions and Execution](06-trading-decisions-and-execution.md) — How the trading engine polls recommendations, runs pre-trade checks, sizes positions, enforces circuit breakers, and submits orders through the broker adapter.
+
+---
+
+## Diagrams
+
+The following Mermaid diagram files can be rendered independently or referenced from the narrative pages:
+
+- [Ingestion to Extraction Flow](diagrams/ingestion-to-extraction-flow.md) — Flowchart from Scheduler through Ingestion, Parser, to Extractor with all queues and storage.
+- [Three-Layer Signal Merging](diagrams/three-layer-signal-merging.md) — Company, Macro, and Competitive signal layers converging into the Aggregation engine.
+- [Weighted Signal Computation](diagrams/weighted-signal-computation.md) — Component breakdown of the composite weight formula.
+- [Trend Accumulation and Escalation](diagrams/trend-accumulation-escalation.md) — How consecutive signals strengthen trends and escalate actions across time windows.
+- [Recommendation Generation Flow](diagrams/recommendation-generation-flow.md) — From TrendSummary through suppression, eligibility, thesis, risk classification, to persistence.
+- [Trading Engine Decision Loop](diagrams/trading-engine-decision-loop.md) — Pre-trade check sequence, position sizing, and order submission flow.
+
+---
+
+## Related Documentation
+
+For reference-level detail on individual services, AI agent configuration, and infrastructure, see the existing documentation:
+
+- [Services Reference](../services.md) — Per-service configuration, database tables, queues, and runtime behaviors.
+- [AI Agents Guide](../ai-agents.md) — AI agent configuration, variants, A/B testing, and the agent management API.
+- [Data Pipeline Architecture](../architecture-data-pipeline.md) — Queue topology, data store summary, and Mermaid flow diagrams for the full data pipeline.
+- [LLM-to-Trade Pipeline](../llm-to-trade-pipeline.md) — End-to-end data flow from model output through signal aggregation to trade execution.
diff --git a/docs/observability.md b/docs/observability.md
new file mode 100644
index 0000000..2343613
--- /dev/null
+++ b/docs/observability.md
@@ -0,0 +1,612 @@
+# Observability and Metrics Reference
+
+This document covers the full observability stack for Stonks Oracle: Prometheus metrics, operational alerting, structured logging, dead-letter queues, and recommended monitoring queries.
+
+## Prometheus Metrics Endpoint
+
+The Query API exposes a `/metrics` endpoint that returns all registered Prometheus metrics in the standard text exposition format.
+
+**Endpoint**: `GET /metrics` on the Query API service (port 8000)
+
+**Response**: `text/plain; version=0.0.4; charset=utf-8` — standard Prometheus scrape format via `prometheus_client.generate_latest()`.
+
+### Prometheus Scrape Configuration
+
+Add the following job to your `prometheus.yml`:
+
+```yaml
+scrape_configs:
+ - job_name: "stonks-oracle"
+ scrape_interval: 15s
+ scrape_timeout: 10s
+ metrics_path: /metrics
+ static_targets:
+ - targets:
+ # Docker Compose
+ - "query-api:8000"
+ # Kubernetes
+ # - "query-api.stonks-oracle.svc.cluster.local:8000"
+```
+
+For Kubernetes deployments, you can also use a `ServiceMonitor` resource if the Prometheus Operator is installed:
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+ name: stonks-oracle
+ namespace: stonks-oracle
+spec:
+ selector:
+ matchLabels:
+ app: query-api
+ endpoints:
+ - port: http
+ path: /metrics
+ interval: 15s
+```
+
+---
+
+## Prometheus Metrics Reference
+
+All metrics are defined in `services/shared/metrics.py`. Metric names use the `stonks_` prefix.
+
+### Service Info
+
+| Metric | Type | Description |
+|--------|------|-------------|
+| `stonks_oracle_info` | Info | Service metadata (build version, etc.) |
+
+### Ingestion Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_ingestion_jobs_total` | Counter | `source_type`, `status` | Total ingestion jobs processed |
+| `stonks_ingestion_items_fetched_total` | Counter | `source_type` | Total items fetched from external sources |
+| `stonks_ingestion_items_new_total` | Counter | `source_type` | New (non-duplicate) items ingested |
+| `stonks_ingestion_items_deduped_total` | Counter | `source_type` | Items skipped due to deduplication |
+| `stonks_ingestion_errors_total` | Counter | `source_type` | Ingestion errors by source type |
+| `stonks_ingestion_adapter_duration_seconds` | Histogram | `source_type` | Adapter fetch latency (buckets: 0.1, 0.5, 1, 2, 5, 10, 30, 60s) |
+
+### Parsing Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_parse_jobs_total` | Counter | `status` | Total parse jobs processed |
+| `stonks_parse_quality_score` | Histogram | — | Distribution of parser quality scores (buckets: 0.1–1.0 in 0.1 steps) |
+| `stonks_parse_low_quality_total` | Counter | — | Documents flagged as low quality by the parser |
+| `stonks_parse_duration_seconds` | Histogram | — | Parse job duration (buckets: 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10s) |
+
+### Extraction Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_extraction_jobs_total` | Counter | `status` | Total extraction jobs processed |
+| `stonks_extraction_attempts_total` | Counter | — | Total Ollama extraction attempts (including retries) |
+| `stonks_extraction_retries_total` | Counter | — | Extraction retry count |
+| `stonks_extraction_duration_seconds` | Histogram | — | Extraction total duration (buckets: 1, 2, 5, 10, 20, 30, 60, 120s) |
+| `stonks_extraction_confidence` | Histogram | — | Distribution of extraction confidence scores (buckets: 0.1–1.0) |
+| `stonks_extraction_validation_errors_total` | Counter | — | Total validation errors across extractions |
+| `stonks_extraction_tokens_total` | Counter | `direction` | Estimated token usage (labels: `input`, `output`) |
+
+### Aggregation Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_aggregation_windows_total` | Counter | `window` | Trend windows computed |
+| `stonks_aggregation_signals_total` | Counter | `window` | Signals processed during aggregation |
+| `stonks_aggregation_contradiction_score` | Histogram | — | Distribution of contradiction scores in trend windows (buckets: 0.0–1.0) |
+| `stonks_aggregation_duration_seconds` | Histogram | `window` | Aggregation job duration (buckets: 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10s) |
+
+### Recommendation Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_recommendations_total` | Counter | `action`, `mode` | Recommendations generated |
+| `stonks_recommendations_suppressed_total` | Counter | — | Recommendations suppressed due to low data quality |
+| `stonks_recommendation_confidence` | Histogram | — | Distribution of recommendation confidence scores (buckets: 0.1–1.0) |
+
+### Lake Publication Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_lake_facts_published_total` | Counter | `table_name` | Analytical facts published to the lakehouse |
+| `stonks_lake_publish_duration_seconds` | Histogram | `table_name` | Lake publication write latency (buckets: 0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5s) |
+| `stonks_lake_publish_errors_total` | Counter | `table_name` | Lake publication errors |
+| `stonks_lake_publish_bytes_total` | Counter | `table_name` | Total bytes written to the lakehouse |
+
+### Trading and Broker Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_orders_submitted_total` | Counter | `side`, `order_type`, `mode` | Orders submitted to broker |
+| `stonks_orders_rejected_total` | Counter | `reason_category` | Orders rejected before broker submission |
+| `stonks_orders_filled_total` | Counter | `side` | Orders filled by broker |
+| `stonks_orders_duplicates_prevented_total` | Counter | `detected_via` | Duplicate orders prevented by idempotency checks |
+| `stonks_risk_evaluations_total` | Counter | `result` | Risk evaluations performed |
+| `stonks_risk_check_failures_total` | Counter | `check_name` | Individual risk check failures |
+| `stonks_positions_synced_total` | Counter | — | Position sync operations completed |
+
+### Alerting Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_alerts_fired_total` | Counter | `rule`, `severity` | Total alerts fired by rule |
+| `stonks_alerts_resolved_total` | Counter | `rule` | Total alerts resolved by rule |
+| `stonks_alert_check_duration_seconds` | Histogram | — | Duration of alert evaluation cycle (buckets: 0.01–5s) |
+| `stonks_alert_active` | Gauge | `rule` | Whether an alert rule is currently firing (1) or resolved (0) |
+
+### Dead-Letter Queue Metrics
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_dlq_items_total` | Counter | `queue` | Jobs sent to dead-letter queues |
+| `stonks_dlq_replayed_total` | Counter | `queue` | Jobs replayed from dead-letter queues |
+| `stonks_dlq_depth` | Gauge | `queue` | Current dead-letter queue depth |
+
+### Active Jobs Gauge
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `stonks_active_jobs` | Gauge | `stage` | Currently processing jobs by pipeline stage |
+
+---
+
+## Alerting Module
+
+The alerting module (`services/shared/alerting.py`) evaluates four operational alert rules against PostgreSQL state on a configurable interval. When a threshold is breached, the module emits structured log events and increments Prometheus counters. When a previously firing alert clears, it logs a resolution event.
+
+### Alert Rules
+
+#### 1. `source_failures` — Sustained Source Retrieval Failures
+
+Detects sources where the last N ingestion runs all failed within the lookback window.
+
+| Parameter | ConfigMap Variable | Default | Description |
+|-----------|--------------------|---------|-------------|
+| Consecutive failure threshold | `ALERT_SOURCE_FAILURE_THRESHOLD` | `3` | Number of consecutive failures before alert fires |
+| Lookback window | `ALERT_SOURCE_FAILURE_WINDOW_HOURS` | `6` hours | How far back to check ingestion_runs |
+
+**Severity**: `warning`
+
+**Query**: Checks `ingestion_runs` for sources where the most recent N runs (within the window) all have `status = 'failed'`.
+
+**Details emitted**: `source_id`, `source_type`, `source_name`, `ticker`, `consecutive_failures`
+
+#### 2. `schema_failure_spike` — Extraction Validation Failure Rate
+
+Detects when the extraction schema validation failure rate exceeds a threshold.
+
+| Parameter | ConfigMap Variable | Default | Description |
+|-----------|--------------------|---------|-------------|
+| Failure rate threshold | `ALERT_SCHEMA_FAILURE_RATE_THRESHOLD` | `0.3` (30%) | Failure rate that triggers the alert |
+| Lookback window | `ALERT_SCHEMA_FAILURE_WINDOW_HOURS` | `1` hour | Window for computing failure rate |
+
+**Severity**: `warning` if rate ≥ 30%, `critical` if rate ≥ 50%
+
+**Query**: Computes `failed / total` from `model_performance_metrics` within the window.
+
+**Details emitted**: `total_extractions`, `failed_extractions`, `failure_rate`, `threshold`, `window_hours`
+
+#### 3. `analytical_lag` — Lake Publication Lag
+
+Detects when lake publication has not completed within the threshold for any table.
+
+| Parameter | ConfigMap Variable | Default | Description |
+|-----------|--------------------|---------|-------------|
+| Lag threshold | `ALERT_LAKE_LAG_THRESHOLD_MINUTES` | `60` minutes | Maximum acceptable time since last successful publish |
+
+**Severity**: `warning`
+
+**Query**: Checks `audit_events` for the most recent successful `lake_publish` event per table, alerts if any are older than the threshold.
+
+**Details emitted**: `table_name`, `last_publish`, `lag_minutes`, `threshold_minutes`
+
+#### 4. `broker_issues` — Consecutive Broker Errors
+
+Detects consecutive broker submission errors (rejections, timeouts, connection failures).
+
+| Parameter | ConfigMap Variable | Default | Description |
+|-----------|--------------------|---------|-------------|
+| Error threshold | `ALERT_BROKER_ERROR_THRESHOLD` | `3` | Consecutive broker errors before alert fires |
+| Lookback window | `ALERT_BROKER_ERROR_WINDOW_HOURS` | `1` hour | Window for checking order_events |
+
+**Severity**: `critical`
+
+**Query**: Counts recent `order_events` with `event_type IN ('broker_error', 'broker_timeout', 'connection_failed')`.
+
+**Details emitted**: `error_count`, `threshold`, `window_hours`
+
+### Evaluation Cycle
+
+The alerting module runs on a configurable interval (default: every 120 seconds, controlled by `ALERT_CHECK_INTERVAL_SECONDS`). Each cycle:
+
+1. Runs all four alert rules against PostgreSQL
+2. Compares results to the current `AlertState` to detect new firings and resolutions
+3. For new firings: increments `stonks_alerts_fired_total`, sets `stonks_alert_active` gauge to 1, logs a `WARNING`
+4. For resolutions: increments `stonks_alerts_resolved_total`, sets `stonks_alert_active` gauge to 0, logs an `INFO`
+5. Records the evaluation duration in `stonks_alert_check_duration_seconds`
+
+Each rule check is wrapped in a try/except so a failure in one rule does not block the others.
+
+### ConfigMap Variables Summary
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `ALERT_SOURCE_FAILURE_THRESHOLD` | `3` | Consecutive source failures before alert |
+| `ALERT_SOURCE_FAILURE_WINDOW_HOURS` | `6` | Source failure lookback window (hours) |
+| `ALERT_SCHEMA_FAILURE_RATE_THRESHOLD` | `0.3` | Extraction failure rate threshold (0.0–1.0) |
+| `ALERT_SCHEMA_FAILURE_WINDOW_HOURS` | `1` | Schema failure lookback window (hours) |
+| `ALERT_LAKE_LAG_THRESHOLD_MINUTES` | `60` | Max minutes since last lake publish |
+| `ALERT_BROKER_ERROR_THRESHOLD` | `3` | Consecutive broker errors before alert |
+| `ALERT_BROKER_ERROR_WINDOW_HOURS` | `1` | Broker error lookback window (hours) |
+| `ALERT_CHECK_INTERVAL_SECONDS` | `120` | Seconds between alert evaluation cycles |
+
+---
+
+## Structured Logging
+
+All services use structured JSON logging configured via `services/shared/logging.py`. Call `setup_logging(service_name)` once at service startup.
+
+### JSON Log Format
+
+Each log line is a single JSON object with the following fields:
+
+```json
+{
+ "timestamp": "2025-01-15T12:34:56.789012+00:00",
+ "level": "INFO",
+ "logger": "ingestion_worker",
+ "message": "Processed job for AAPL",
+ "service": "ingestion_worker",
+ "trace_id": "a1b2c3d4e5f67890",
+ "span_id": "1a2b3c4d"
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `timestamp` | string (ISO 8601) | UTC timestamp of the log event |
+| `level` | string | Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
+| `logger` | string | Python logger name |
+| `message` | string | Human-readable log message |
+| `service` | string | Service name set at startup (e.g., `ingestion_worker`, `scheduler`) |
+| `trace_id` | string | 16-character hex trace ID for distributed tracing |
+| `span_id` | string | 8-character hex span ID for the current operation |
+
+### Additional Context Fields
+
+When present, these fields are merged into the JSON output:
+
+| Field | Source | Description |
+|-------|--------|-------------|
+| `span_operation` | `Span` context manager | Name of the traced operation |
+| `span_status` | `Span` context manager | `ok` or `error` |
+| `span_duration_ms` | `Span` context manager | Duration of the span in milliseconds |
+| `span_parent_id` | `Span` context manager | Parent span ID for nested spans |
+| `span_attributes` | `Span` context manager | Arbitrary key-value attributes set on the span |
+| `ticker` | Manual `extra={}` | Company ticker symbol |
+| `document_id` | Manual `extra={}` | Document UUID |
+| `source_type` | Manual `extra={}` | Source type (e.g., `polygon`, `news_api`) |
+| `job_id` | Manual `extra={}` | Job identifier |
+| `duration_ms` | Manual `extra={}` | Operation duration |
+| `error` | Manual `extra={}` | Error description |
+| `count` | Manual `extra={}` | Item count |
+| `exception` | Automatic | Formatted exception traceback (when `exc_info` is set) |
+
+### Trace Context Propagation
+
+Trace context flows through the pipeline via job payloads:
+
+1. **Inject**: Before enqueuing a job to Redis, call `inject_trace_context(payload)` to add `_trace_id` to the payload dict.
+2. **Extract**: At the start of job processing, call `extract_trace_context(payload)` to restore the trace context (or generate a new one if absent).
+3. **Span**: Use the `Span` context manager to create child spans within a service:
+
+```python
+from services.shared.logging import Span
+
+with Span("process_document", ticker="AAPL") as span:
+ # ... do work ...
+ span.set_attribute("doc_count", 5)
+```
+
+This produces a structured log entry on span exit with duration, status, and attributes.
+
+### Log Querying
+
+To trace a request through the pipeline, filter by `trace_id`:
+
+```bash
+# Kubernetes — find all logs for a specific trace
+kubectl logs -n stonks-oracle -l app.kubernetes.io/part-of=stonks-oracle --all-containers \
+ | jq -r 'select(.trace_id == "a1b2c3d4e5f67890")'
+
+# Docker Compose — search across all services
+docker compose logs --no-color | grep '"trace_id":"a1b2c3d4e5f67890"'
+```
+
+To find errors in a specific service:
+
+```bash
+# Kubernetes
+kubectl logs -n stonks-oracle deployment/extractor --tail=500 \
+ | jq 'select(.level == "ERROR")'
+
+# Docker Compose
+docker compose logs extractor --no-color --tail=500 \
+ | jq 'select(.level == "ERROR")'
+```
+
+To find slow extraction spans:
+
+```bash
+kubectl logs -n stonks-oracle deployment/extractor --tail=1000 \
+ | jq 'select(.span_operation == "extract_document" and .span_duration_ms > 30000)'
+```
+
+---
+
+## Dead-Letter Queue System
+
+When a worker fails to process a job after exhausting retries (default: 3 attempts), the job is pushed to a per-queue dead-letter list in Redis. The DLQ system is implemented in `services/shared/dead_letter.py`.
+
+### Queue Names
+
+Dead-letter queues follow the naming pattern `stonks:dlq:<queue_name>`:
+
+| DLQ Key | Source Queue | Description |
+|---------|-------------|-------------|
+| `stonks:dlq:ingestion` | `stonks:queue:ingestion` | Failed ingestion jobs (adapter errors, API failures) |
+| `stonks:dlq:parsing` | `stonks:queue:parsing` | Failed parse jobs |
+| `stonks:dlq:extraction` | `stonks:queue:extraction` | Failed extraction jobs (LLM errors, validation failures) |
+| `stonks:dlq:aggregation` | `stonks:queue:aggregation` | Failed aggregation jobs |
+| `stonks:dlq:recommendation` | `stonks:queue:recommendation` | Failed recommendation jobs |
+| `stonks:dlq:broker_orders` | `stonks:queue:broker_orders` | Failed broker order submissions |
+
+When `DEPLOY_STAGE` is set, the prefix becomes `stonks:<stage>:dlq:<queue_name>`.
+
+### DLQ Entry Format
+
+Each DLQ entry wraps the original job payload with failure metadata:
+
+```json
+{
+ "original_payload": {
+ "source_id": "...",
+ "source_type": "polygon",
+ "ticker": "AAPL",
+ "company_id": "...",
+ "config": {}
+ },
+ "queue": "ingestion",
+ "error": "ConnectionError: API timeout after 30s",
+ "attempt": 3,
+ "worker": "ingestion_worker",
+ "dead_lettered_at": "2025-01-15T12:34:56.789012+00:00"
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `original_payload` | object | The original job payload as it was enqueued |
+| `queue` | string | Source queue name |
+| `error` | string | Error message from the final failed attempt |
+| `attempt` | integer | Number of attempts made before dead-lettering |
+| `worker` | string | Worker identifier that dead-lettered the job |
+| `dead_lettered_at` | string (ISO 8601) | UTC timestamp when the job was dead-lettered |
+
+### Routing
+
+Jobs are routed to the DLQ by calling `send_to_dlq()` from worker code after retry exhaustion:
+
+```python
+from services.shared.dead_letter import send_to_dlq
+
+await send_to_dlq(
+ rds=redis_client,
+ queue_name="ingestion",
+ original_payload=job,
+ error=str(exception),
+ attempt=3,
+ worker="ingestion_worker",
+)
+```
+
+The default maximum attempts before dead-lettering is `DEFAULT_MAX_ATTEMPTS = 3`.
+
+### Replay Tooling
+
+The `services/shared/dead_letter.py` module provides functions for inspecting and replaying DLQ items:
+
+| Function | Description |
+|----------|-------------|
+| `peek_dlq(rds, queue_name, start=0, count=10)` | Inspect DLQ entries without removing them |
+| `replay_one(rds, queue_name)` | Pop the oldest DLQ entry and re-enqueue its original payload to the source queue |
+| `replay_all(rds, queue_name)` | Replay every item in the DLQ back to the source queue. Returns the count replayed |
+| `dlq_length(rds, queue_name)` | Return the number of items in the DLQ |
+| `dlq_summary(rds, queue_names)` | Return a mapping of queue_name → DLQ depth for multiple queues |
+| `purge_dlq(rds, queue_name)` | Delete all items from the DLQ. Returns count removed |
+
+### Monitoring DLQ Depth
+
+Use the `scripts/check_queues.py` script to inspect queue and DLQ depths from the command line:
+
+```bash
+# Docker Compose
+REDIS_HOST=localhost REDIS_PORT=6379 REDIS_PASSWORD="" \
+ python scripts/check_queues.py
+
+# Kubernetes
+kubectl exec -n stonks-oracle deployment/query-api -- \
+ python scripts/check_queues.py
+```
+
+The Query API also exposes DLQ depths in the `/api/ops/pipeline/stream` SSE endpoint and the DevOps metrics endpoints, reporting `dlq:<queue_name>` keys alongside regular queue depths.
+
+The `stonks_dlq_depth` Prometheus gauge tracks DLQ depth per queue for dashboard alerting.
+
+---
+
+## Recommended Prometheus/Grafana Queries
+
+### Ingestion Throughput
+
+```promql
+# Ingestion jobs per minute by source type and status
+sum(rate(stonks_ingestion_jobs_total[5m])) by (source_type, status) * 60
+
+# New items ingested per minute
+sum(rate(stonks_ingestion_items_new_total[5m])) * 60
+
+# Deduplication ratio (higher = more duplicates being filtered)
+sum(rate(stonks_ingestion_items_deduped_total[5m]))
+ / sum(rate(stonks_ingestion_items_fetched_total[5m]))
+
+# Adapter latency p95 by source type
+histogram_quantile(0.95, sum(rate(stonks_ingestion_adapter_duration_seconds_bucket[5m])) by (le, source_type))
+
+# Ingestion error rate
+sum(rate(stonks_ingestion_errors_total[5m])) by (source_type)
+```
+
+### Extraction Latency and Quality
+
+```promql
+# Extraction duration p50 and p95
+histogram_quantile(0.5, sum(rate(stonks_extraction_duration_seconds_bucket[5m])) by (le))
+histogram_quantile(0.95, sum(rate(stonks_extraction_duration_seconds_bucket[5m])) by (le))
+
+# Extraction success rate
+sum(rate(stonks_extraction_jobs_total{status="success"}[5m]))
+ / sum(rate(stonks_extraction_jobs_total[5m]))
+
+# Average extraction confidence
+histogram_quantile(0.5, sum(rate(stonks_extraction_confidence_bucket[5m])) by (le))
+
+# Validation error rate
+sum(rate(stonks_extraction_validation_errors_total[5m]))
+
+# Token usage rate (input vs output)
+sum(rate(stonks_extraction_tokens_total[5m])) by (direction)
+```
+
+### Aggregation Volume
+
+```promql
+# Trend windows computed per minute by window size
+sum(rate(stonks_aggregation_windows_total[5m])) by (window) * 60
+
+# Signals processed per minute
+sum(rate(stonks_aggregation_signals_total[5m])) by (window) * 60
+
+# Average contradiction score (higher = more conflicting signals)
+histogram_quantile(0.5, sum(rate(stonks_aggregation_contradiction_score_bucket[5m])) by (le))
+
+# Aggregation duration p95
+histogram_quantile(0.95, sum(rate(stonks_aggregation_duration_seconds_bucket[5m])) by (le, window))
+```
+
+### Recommendation Generation
+
+```promql
+# Recommendations generated per minute by action
+sum(rate(stonks_recommendations_total[5m])) by (action, mode) * 60
+
+# Suppression rate
+sum(rate(stonks_recommendations_suppressed_total[5m]))
+ / sum(rate(stonks_recommendations_total[5m]))
+
+# Recommendation confidence distribution
+histogram_quantile(0.5, sum(rate(stonks_recommendation_confidence_bucket[5m])) by (le))
+```
+
+### Trading Engine Activity
+
+```promql
+# Orders submitted per minute by side
+sum(rate(stonks_orders_submitted_total[5m])) by (side, mode) * 60
+
+# Order rejection rate by reason
+sum(rate(stonks_orders_rejected_total[5m])) by (reason_category)
+
+# Fill rate
+sum(rate(stonks_orders_filled_total[5m]))
+ / sum(rate(stonks_orders_submitted_total[5m]))
+
+# Duplicate orders prevented
+sum(rate(stonks_orders_duplicates_prevented_total[5m])) by (detected_via)
+
+# Risk evaluation outcomes
+sum(rate(stonks_risk_evaluations_total[5m])) by (result)
+
+# Risk check failure breakdown
+sum(rate(stonks_risk_check_failures_total[5m])) by (check_name)
+```
+
+### Lake Publication
+
+```promql
+# Facts published per minute by table
+sum(rate(stonks_lake_facts_published_total[5m])) by (table_name) * 60
+
+# Write latency p95 by table
+histogram_quantile(0.95, sum(rate(stonks_lake_publish_duration_seconds_bucket[5m])) by (le, table_name))
+
+# Publication error rate
+sum(rate(stonks_lake_publish_errors_total[5m])) by (table_name)
+
+# Bytes written per minute
+sum(rate(stonks_lake_publish_bytes_total[5m])) by (table_name) * 60
+```
+
+### Alerting Health
+
+```promql
+# Currently active alerts by rule
+stonks_alert_active
+
+# Alert firing rate
+sum(rate(stonks_alerts_fired_total[1h])) by (rule, severity)
+
+# Alert evaluation duration
+histogram_quantile(0.95, sum(rate(stonks_alert_check_duration_seconds_bucket[5m])) by (le))
+```
+
+### Dead-Letter Queue Health
+
+```promql
+# Current DLQ depth by queue
+stonks_dlq_depth
+
+# DLQ inflow rate (jobs dead-lettered per minute)
+sum(rate(stonks_dlq_items_total[5m])) by (queue) * 60
+
+# DLQ replay rate
+sum(rate(stonks_dlq_replayed_total[5m])) by (queue) * 60
+```
+
+### Pipeline Overview (Active Jobs)
+
+```promql
+# Currently active jobs by pipeline stage
+stonks_active_jobs
+
+# Parse quality score distribution
+histogram_quantile(0.5, sum(rate(stonks_parse_quality_score_bucket[5m])) by (le))
+
+# Low quality document rate
+sum(rate(stonks_parse_low_quality_total[5m]))
+ / sum(rate(stonks_parse_jobs_total[5m]))
+```
+
+### Recommended Grafana Alert Rules
+
+| Alert | Expression | For | Severity |
+|-------|-----------|-----|----------|
+| High DLQ depth | `stonks_dlq_depth > 10` | 5m | warning |
+| Ingestion error spike | `sum(rate(stonks_ingestion_errors_total[5m])) > 0.5` | 5m | warning |
+| Extraction latency high | `histogram_quantile(0.95, sum(rate(stonks_extraction_duration_seconds_bucket[5m])) by (le)) > 60` | 10m | warning |
+| Lake publication stale | `stonks_alert_active{rule="analytical_lag"} == 1` | 5m | warning |
+| Broker errors active | `stonks_alert_active{rule="broker_issues"} == 1` | 1m | critical |
+| Zero ingestion throughput | `sum(rate(stonks_ingestion_jobs_total[15m])) == 0` | 15m | critical |
diff --git a/docs/sanitized-pipeline-deep-dive/01-data-ingestion-and-preparation.md b/docs/sanitized-pipeline-deep-dive/01-data-ingestion-and-preparation.md
new file mode 100644
index 0000000..daad3e8
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/01-data-ingestion-and-preparation.md
@@ -0,0 +1,130 @@
+# Page 1 — Data Ingestion and Preparation
+
+Every signal that the platform eventually acts on begins its life as raw data pulled from an external source. Before any AI agent can extract structured intelligence, before any trend can accumulate, and before any decision can be executed, the platform must first discover new content, fetch it reliably, eliminate duplicates, store the raw artifacts for audit, and normalize the text into a form suitable for downstream processing. This page traces that journey from external API to parser output, covering the Scheduler, Ingestion Worker, deduplication layer, raw storage, and Parser in detail.
+
+For a visual overview of the full flow described here, see the [Ingestion to Extraction Flow diagram](diagrams/ingestion-to-extraction-flow.md).
+
+---
+
+## Four Categories of Input Data
+
+The platform tracks 50 entities across 10 sectors, and it draws intelligence from four distinct categories of external data. Each category has its own adapter, its own API conventions, and its own scheduling cadence, but all of them feed into the same ingestion pipeline.
+
+The first category is **entity news**, sourced from the external data provider's news endpoint (`/v2/reference/news`). The `ExternalNewsAdapter` in `services/adapters/news_adapter.py` fetches articles linked to a specific entity identifier, returning structured results that include title, publisher, article URL, description, keywords, and publication timestamp. Each request can return up to 1,000 articles, though the default limit is 20 per fetch. The adapter tracks the most recent `published_utc` value and uses it on subsequent fetches to avoid re-retrieving articles the system has already seen.
+
+The second category is **regulatory filings**, sourced from the public records API full-text search system (regulatory filings source). The `RegulatoryFilingsAdapter` in `services/adapters/filings_adapter.py` queries the `/LATEST/search-index` endpoint for regulatory filing types and other form types associated with an entity's identifier or CIK number. Unlike the external data provider endpoints, the public records API requires no key — only a descriptive `User-Agent` header per the API's fair-access policy. The adapter deduplicates results by accession number (`adsh`), filters out non-primary documents like XML fragments and graphics, and constructs the public records API filing index URL for each hit so downstream services can fetch the full document.
+
+The third category is **data feeds**, also sourced from the external data provider. The `ExternalDataAdapter` in `services/adapters/market_adapter.py` supports multiple endpoints: previous-day aggregate bars (`/v2/aggs/ticker/{ticker}/prev`), range bars for custom date windows, intraday hourly bars, grouped daily bars that return data for all entities in a single call (`/v2/aggs/grouped/locale/us/market/stocks/{date}`), and entity detail lookups. Data feeds follow a different path than textual content — they do not pass through the Parser or Extractor, since the structured numeric data is already in a usable form.
+
+The fourth category is **macro and geopolitical news**, fetched by the `MacroNewsAdapter` in `services/adapters/macro_news_adapter.py`. Unlike the other three categories, macro news is not entity-specific. These sources have `source_type='macro_news'` in the `sources` database table and may have a `NULL` `company_id`. The adapter fetches from a configurable HTTP endpoint (typically the external data provider's news API filtered for broad topics) and returns articles that describe global events — policy shifts, central bank decisions, geopolitical conflicts — rather than entity-specific developments. Macro news articles are eventually classified by the Global Event Classifier agent and routed through a separate queue, as described in [Page 2](02-ai-agent-processing-and-extraction.md).
+
+All four adapter classes inherit from `BaseAdapter` defined in `services/adapters/base.py` and return an `AdapterResult` dataclass containing the raw payload bytes, a SHA-256 content hash, a list of parsed item dicts, HTTP metadata (status code, response time), and an error field that is `None` on success. This uniform interface allows the Ingestion Worker to handle all source types through a single dispatch mechanism.
+
+---
+
+## The Scheduler: Orchestrating Ingestion Cycles
+
+The Scheduler (`services/scheduler/app.py`) is the heartbeat of the ingestion pipeline. It runs a continuous loop that ticks every 15 seconds (`SCHEDULER_TICK = 15`), and on each tick it evaluates which sources are due for their next fetch. The Scheduler does not fetch data itself — it enqueues jobs onto the `app:queue:ingestion` Redis list for the Ingestion Worker to process.
+
+Each source type has a default polling cadence defined in the `DEFAULT_CADENCES` dictionary:
+
+| Source Type | Default Cadence |
+|------------------|-----------------|
+| `market_api` | 300 seconds |
+| `news_api` | 300 seconds |
+| `filings_api` | 3,600 seconds |
+| `macro_news` | 600 seconds |
+| `web_scrape` | 1,800 seconds |
+| `execution_api` | 30 seconds |
+
+Individual sources can override their cadence via the `polling_interval_seconds` field in their `config` JSONB column in the `sources` table. The `get_cadence_for_source()` function checks for this override first, falling back to the default if none is set, and enforces a minimum interval of 10 seconds.
+
+The Scheduler determines whether a source is due by calling `is_source_due()`, which considers several conditions. If a source has never run before (no entry in the `ingestion_runs` table), it is immediately due. If the last run failed, the Scheduler respects an exponential backoff computed by `compute_backoff()`: the delay starts at 60 seconds (`DEFAULT_BACKOFF_BASE`) and doubles with each retry up to a maximum of 3,600 seconds (`MAX_BACKOFF`). If a source has failed 10 consecutive times (`MAX_RETRY_COUNT`), the Scheduler stops scheduling it entirely until an operator manually resets the retry state. If the last run is still marked as `running`, the source is skipped to prevent double-scheduling. Otherwise, the Scheduler checks whether enough time has elapsed since the last completed run based on the source's cadence.
+
+Rate limiting adds another layer of protection. The `check_rate_limit()` function enforces two constraints. First, each source type has a per-type limit defined in `DEFAULT_RATE_LIMITS` — for example, `market_api` and `news_api` are each capped at 20 requests per minute, while `filings_api` and `macro_news` are capped at 10. Second, because `market_api` and `news_api` both use the same external data provider API key, a global provider rate limit of 45 requests per minute (`PROVIDER_GLOBAL_RATE_LIMIT`) is enforced across both types combined. Rate limit state is tracked in Redis using keys of the form `app:ratelimit:{source_type}:{window}`, where the window is a minute-granularity timestamp. If a source type exceeds its limit, the Scheduler logs a warning and skips that source for the current tick.
+
+The Scheduler handles three categories of sources in each cycle. First, it fetches all active entity-specific sources (excluding `macro_news`) by joining the `sources` and `companies` tables. Second, it fetches active macro news sources separately, since these may not have a `company_id`. Third, it fetches global data sources — those with `source_type='market_api'` and `company_id IS NULL` — which represent endpoints like the grouped daily bars that return data for all entities in a single API call. For intraday bar sources, the Scheduler expands a single global source into per-entity jobs for every active entity.
+
+Each enqueued job payload includes the `source_id`, `company_id`, `ticker`, `legal_name`, `source_type`, `source_name`, `config`, `credibility_score`, a list of company `aliases` (fetched from the `company_aliases` table), and a `scheduled_at` timestamp. The job is pushed onto `app:queue:ingestion` via Redis `RPUSH`.
+
+Beyond scheduling, the Scheduler also performs periodic maintenance. Every ~20 cycles (~5 minutes), it runs `recover_stale_documents()` to re-enqueue documents that have been stuck in `parsed` status for longer than 240 minutes — a safety net for cases where Redis loses queue entries due to pod restarts or OOM events. Every ~40 cycles (~10 minutes), it runs `retry_failed_extractions()` to give documents in `extraction_failed` status another chance, resetting them to `parsed` and deleting the failed `document_intelligence` row so the Extractor treats them as fresh. Every ~100 cycles (~25 minutes), it runs `cleanup_all_tables()` to enforce retention policies across tables like `competitive_signal_records` (30 days), `ingestion_runs` (14 days), and `execution_decisions` (90 days).
+
+For more detail on the Scheduler's configuration and operational behavior, see the [Services Reference](../services.md).
+
+---
+
+## The Ingestion Worker: Adapter Dispatch and Persistence
+
+The Ingestion Worker (`services/ingestion/worker.py`) is a long-running process that continuously pops jobs from the `app:queue:ingestion` Redis list and processes them. On startup, it initializes one instance of each adapter class and stores them in a dispatch dictionary keyed by `source_type`:
+
+```
+adapters = {
+ "market_api": ExternalDataAdapter(...),
+ "news_api": ExternalNewsAdapter(...),
+ "filings_api": RegulatoryFilingsAdapter(),
+ "web_scrape": WebScrapeAdapter(),
+ "execution_api": ExecutionAdapter(...),
+ "macro_news": MacroNewsAdapter(...),
+}
+```
+
+When a job arrives, the `process_job()` function looks up the appropriate adapter by `source_type` and calls its `fetch()` method with the ticker and source config. Before fetching, it records a new row in the `ingestion_runs` table with status `running`. If the adapter returns an error, the worker calls `record_retrieval_failure()` to update the run status and increment the source's retry counter with exponential backoff timing.
+
+On a successful fetch, the worker performs several steps in sequence. First, it uploads the raw payload to MinIO via `upload_raw_artifact()` in `services/shared/storage.py`. The target bucket is determined by the source type through the `SOURCE_BUCKET_MAP`: `market_api` payloads go to `app-raw-data`, `news_api` and `macro_news` payloads go to `app-raw-content`, and `filings_api` payloads go to `app-raw-filings`. Objects are stored under a path that encodes the source type, entity identifier, date hierarchy, and document ID — for example, `news_api/Entity-A/2025/01/15/{run_id}/raw.json`.
+
+---
+
+## Content Deduplication via Redis
+
+After storing the raw artifact, the Ingestion Worker checks for duplicate content. Deduplication operates at two levels.
+
+At the payload level, the worker checks the overall `content_hash` (a SHA-256 digest of the raw API response) against Redis. The key pattern is `app:dedupe:{content_hash}` with a 24-hour TTL (86,400 seconds). If the hash is already present, the entire payload is skipped — the `ingestion_runs` row is marked as completed with `items_new=0`, and no downstream jobs are enqueued. If the hash is new, the worker sets the marker in Redis so future fetches of identical content are caught.
+
+At the individual item level, for source types other than `market_api` and `execution_api`, the worker calls `dedupe_items()` from `services/shared/dedupe.py`. This function checks each item against a layered deduplication strategy. The fast path checks Redis for both content-hash markers (`app:dedupe:{hash}`) and canonical-URL markers (`app:dedupe:url:{url_hash}`), both with 24-hour TTLs. If the Redis check misses, the function falls back to PostgreSQL, querying the `documents` table by `content_hash` or `canonical_url` for durable cross-source matching. When a duplicate is found through the PostgreSQL fallback, the function warms the Redis cache so subsequent checks are fast.
+
+Items identified as duplicates are not discarded entirely. If the duplicate document was originally ingested for a different entity, the worker creates a cross-source mention link in the `document_company_mentions` table via `persist_document_company_mention()`. This ensures that a news article mentioning both Entity-A and Entity-F is linked to both entities even if it was first ingested through Entity-A's news source.
+
+New (non-duplicate) items are persisted to PostgreSQL through `persist_ingestion_items()` in `services/shared/metadata.py`, which inserts rows into the `documents` table and records entity mentions in `document_company_mentions`. Each new document ID is then pushed onto `app:queue:parsing` for the Parser to process. After persistence, the worker calls `mark_as_seen()` to set Redis dedupe markers for both the content hash and canonical URL of each new item, ensuring that the next fetch cycle's deduplication checks are fast.
+
+On successful completion, the worker updates the `ingestion_runs` row with the final counts (`items_fetched`, `items_new`) and calls `reset_source_retry_state()` to clear any accumulated backoff from previous failures. For news-type sources (`news_api` and `macro_news`), the worker also updates the source's `config` JSONB column with the latest `published_utc` value, so the next fetch only retrieves newer articles.
+
+---
+
+## The Parser: Normalization, Quality Scoring, and Routing
+
+Documents that pass through ingestion arrive on the `app:queue:parsing` Redis list as JSON payloads containing a `document_id`, `ticker`, and `source_type`. The Parser Worker (`services/parser/worker.py`) pops these jobs and transforms raw HTML or text into normalized, quality-scored documents ready for AI extraction.
+
+The parsing pipeline begins with HTML fetching. If the document has a URL (looked up from the `documents` table if not present in the job payload), the worker calls `fetch_html()` to retrieve the page content. Public records API URLs receive a specialized `User-Agent` header to comply with the API's fair-access policy. The raw HTML is then passed to `parse_html()` in `services/parser/html_parser.py`, which runs a multi-stage extraction pipeline.
+
+The HTML parser first strips non-content tags — `script`, `style`, `nav`, `footer`, `header`, `aside`, `iframe`, and others — and removes boilerplate containers identified by CSS class or ID patterns (sidebars, ad slots, newsletter signups, social share bars, and similar UI elements). It then searches for the article body using a priority list of semantic selectors (`article`, `[role='main']`, `.article-body`, `.post-content`, and others). If no semantic match is found, it falls back to text-density scoring across candidate `div`, `section`, and `td` elements, selecting the block with the highest composite score based on text density, link density, paragraph count, and word count. The extracted text undergoes further cleaning: regex-based removal of residual boilerplate phrases (copyright notices, "subscribe to our newsletter" prompts, "share this article" fragments), removal of short orphan lines that are likely UI fragments, detection and collapse of repeated template blocks, and whitespace normalization.
+
+Metadata extraction pulls the document title (from `og:title` or `<title>`), author, publisher (from `og:site_name` or hostname), publication date (from `article:published_time` or JSON-LD `datePublished`), canonical URL, language, description, and keywords from the HTML head elements.
+
+If the parsed body text is shorter than 500 characters, the worker attempts to enrich it by reading the raw API payload from MinIO and extracting the data provider's article description, keywords, and author fields for the matching article. This enrichment step ensures that even articles with minimal scrapeable HTML still have enough textual content for meaningful AI extraction.
+
+Quality scoring is performed by `score_parse_quality()` in `services/parser/html_parser.py`, which evaluates six weighted signals to produce a composite score between 0 and 0.95:
+
+| Signal | Weight | What It Measures |
+|--------------------|--------|-----------------------------------------------------------------|
+| `word_count` | 0.30 | Length of extracted text (thresholds at 20, 50, 150, 300 words) |
+| `body_found` | 0.20 | Whether a semantic article body element was located |
+| `diversity` | 0.15 | Vocabulary richness (unique words / total words) |
+| `sentence` | 0.15 | Presence of proper sentence structure (terminal punctuation) |
+| `paragraph` | 0.10 | Multi-paragraph structure (blocks separated by blank lines) |
+| `metadata` | 0.10 | Presence of title, author, publisher, and publication date |
+
+The composite score maps to a confidence label: scores below 0.35 are labeled `low`, scores between 0.35 and 0.65 are `medium`, and scores 0.65 and above are `high`. Documents with `low` confidence are marked with status `low_quality` in the `documents` table and are not enqueued for extraction — they are effectively filtered out of the pipeline at this stage.
+
+Entity mention detection runs next. The worker fetches all known aliases from the `company_aliases` table (plus entity identifiers and legal names from the `companies` table) and calls `detect_company_mentions()` in `services/parser/html_parser.py`. The matching strategy varies by alias length: one-to-two character aliases use case-sensitive word-boundary matching to avoid false positives (the letter "A" should not match every occurrence of the word "a"), three-to-four character aliases use case-insensitive word-boundary matching (standard identifier format), and aliases of five or more characters use case-insensitive substring matching (entity names and brands). Confidence scores vary by alias type: identifier matches receive 0.9, legal name matches 0.85, general aliases 0.7, and brand matches 0.6. Multiple alias hits for the same entity are deduplicated, keeping the highest-confidence match and summing match counts. Detected mentions are persisted to the `document_company_mentions` table.
+
+The normalized text and a structured parser output JSON (containing all metadata, quality signals, warnings, outbound links, tags, and mentions) are uploaded to the `app-normalized` MinIO bucket. The `documents` row is updated with the normalized storage reference, parser output reference, quality score, and confidence level.
+
+Finally, the Parser makes a routing decision. If the document's `document_type` is `macro_event`, it is pushed onto `app:queue:macro_classification` for the Global Event Classifier agent. All other documents are pushed onto `app:queue:extraction` for the Document Intelligence Extractor agent. Both queues feed into the Extractor service described in [Page 2](02-ai-agent-processing-and-extraction.md). The job payload includes the `document_id`, `ticker`, and the first 32,000 characters of the normalized text, giving the downstream agent immediate access to the content without needing to fetch it from MinIO.
+
+For additional detail on queue topology and data store layout, see the [Data Pipeline Architecture](../architecture-data-pipeline.md) documentation.
+
+---
+
+## What Comes Next
+
+At this point, raw data has been fetched from four external sources, deduplicated, stored in MinIO, parsed into normalized text, scored for quality, tagged with entity mentions, and routed to the appropriate extraction queue. The documents sitting on `app:queue:extraction` and `app:queue:macro_classification` are clean, quality-filtered, and ready for AI processing. [Page 2 — AI Agent Processing and Structured Extraction](02-ai-agent-processing-and-extraction.md) picks up the story from here, explaining how the Document Intelligence Extractor and Global Event Classifier agents use LLM inference to transform these normalized documents into the structured JSON intelligence that feeds the rest of the pipeline.
diff --git a/docs/sanitized-pipeline-deep-dive/02-ai-agent-processing-and-extraction.md b/docs/sanitized-pipeline-deep-dive/02-ai-agent-processing-and-extraction.md
new file mode 100644
index 0000000..5cf3c3e
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/02-ai-agent-processing-and-extraction.md
@@ -0,0 +1,164 @@
+# Page 2 — AI Agent Processing and Structured Extraction
+
+Documents that arrive on the `app:queue:extraction` and `app:queue:macro_classification` Redis queues are clean, quality-filtered, and normalized — but they are still unstructured text. The job of the Extractor service is to transform that text into structured JSON intelligence that the rest of the pipeline can reason about quantitatively. Two AI agents share this responsibility: the Document Intelligence Extractor handles entity-specific content, filings, and transcripts, while the Global Event Classifier handles macro-level geopolitical and economic events. Both agents run through the same Ollama-based inference infrastructure, share a common JSON repair pipeline, and persist their results to PostgreSQL and MinIO for downstream consumption and audit.
+
+This page explains how each agent works, what schemas they produce, how the system validates and repairs LLM output, how runtime configuration is resolved from the database, and how the final structured records are persisted. For a visual overview of the full flow from ingestion through extraction, see the [Ingestion to Extraction Flow diagram](diagrams/ingestion-to-extraction-flow.md). For reference-level detail on agent configuration and the variant management API, see the [AI Agents Guide](../ai-agents.md).
+
+---
+
+## The Document Intelligence Extractor
+
+The Document Intelligence Extractor is the primary AI agent in the pipeline. Registered under the slug `document-extractor` in the `ai_agents` database table, it processes every non-macro document that passes through the Parser — news articles, regulatory filings, performance transcripts, and press releases. Its purpose is to read a normalized document and produce a structured JSON object that captures the document's summary, the entities it affects, the sentiment and impact for each entity, the catalysts driving that impact, and the evidence supporting the analysis.
+
+The entry point is `services/extractor/main.py`, which runs a continuous worker loop polling the `app:queue:extraction` Redis list. When a job arrives, the worker extracts the `document_id`, `ticker`, and `text` fields from the JSON payload. If the job payload does not include the document text directly, the worker fetches it from MinIO using the `normalized_storage_ref` stored in the `documents` table — the Parser uploaded the normalized text to the `app-normalized` bucket during the previous pipeline stage (see [Page 1](01-data-ingestion-and-preparation.md)).
+
+The actual LLM inference is handled by `OllamaClient` in `services/extractor/client.py`. The client sends the document to a local Ollama instance via the `/api/chat` HTTP endpoint with `stream=False` and `think=False`. The `think=False` flag is a deliberate performance choice — it disables the model's chain-of-thought reasoning phase, which would otherwise add two to four minutes of latency per document. The client does not use Ollama's `format` parameter for structured output because of a known Ollama bug (#14645) where the format constraint is silently ignored when `think=False` on qwen3.5 models. Instead, the system relies on prompt engineering to produce JSON and repairs any syntax issues after the fact.
+
+The prompt sent to the model has two parts. The system prompt, defined in `services/extractor/prompts.py`, establishes the model's role as a document analyst and sets strict output rules: return only a single JSON object, no markdown fences, no explanation text, every schema field is required, use `"other"` for `catalyst_type` when unsure, keep evidence spans under 20 words, and limit key facts to three to five items. The user prompt, built by `build_extraction_prompt()` in the same module, provides the document text along with document-type-specific guidance. Four guidance variants exist — one each for articles, filings, transcripts, and press releases — each calibrated to the conventions and biases of that document type. For example, the filing guidance instructs the model to preserve the precise legal language of regulatory documents, while the press release guidance warns that sentiment may be biased positive and directs the model to focus on concrete metrics rather than marketing language.
+
+The user prompt also includes a list of all tracked entity identifiers from the `companies` table, along with rules for how the model should use them. If a tracked entity identifier appears verbatim in the text, the model must include it in the output with at least one evidence span. If the article discusses a sector or theme that clearly affects a tracked entity (oil prices affecting Entity-D, AI chip demand affecting Entity-C), the model should include that entity as well. The model is explicitly told not to invent identifiers that are not in the provided list. Documents longer than 8,000 characters are truncated before being included in the prompt, with a `[... truncated for extraction ...]` marker appended.
+
+The `OllamaClient` also supports a `context_window` override via the Ollama `num_ctx` option, which can be configured per agent variant through the `AgentConfigResolver` mechanism described later in this page.
+
+---
+
+## The ExtractionResult Schema
+
+The structured output that the Document Intelligence Extractor produces is defined by the `ExtractionResult` Pydantic model in `services/extractor/schemas.py`. Every field is required — the model has no defaults — so the generated JSON schema forces the LLM to produce every field explicitly. The top-level fields are:
+
+**`summary`** — a concise one-to-three sentence summary of the document's main point. This becomes the human-readable description stored in the `document_intelligence` table.
+
+**`companies`** — an array of `CompanyExtractionItem` objects, one per affected entity. Each entity entry contains:
+
+- `ticker` — the entity identifier (validated against a regex pattern of one to five uppercase letters).
+- `company_name` — the full entity name as referenced in the document.
+- `relevance` — a float between 0.0 and 1.0 indicating how relevant the document is to this entity, where 0 means tangential and 1 means the entity is the primary subject.
+- `sentiment` — one of `positive`, `negative`, `neutral`, or `mixed`, representing the overall sentiment toward this entity in the document.
+- `impact_score` — a float between 0.0 and 1.0 estimating the magnitude of impact, where 0 is negligible and 1 is highly material.
+- `impact_horizon` — one of `intraday`, `1d`, `1d_7d`, `1d_30d`, `30d_90d`, or `90d_plus`, indicating the expected timeframe over which the impact will play out.
+- `catalyst_type` — exactly one of `performance_report`, `product`, `legal`, `macro`, `supply_chain`, `m_and_a`, `rating_change`, or `other`. The prompt instructs the model to use `other` when none of the specific categories fit.
+- `key_facts` — a list of facts explicitly stated in the document. The prompt emphasizes that the model must not infer or fabricate facts.
+- `risks` — a list of risks explicitly mentioned in the document.
+- `evidence_spans` — short verbatim quotes from the document supporting the analysis. The prompt requests these be kept under 20 words each.
+
+**`macro_themes`** — a list of broad economic or environmental themes mentioned in the document, such as `rates`, `inflation`, or `ai_capex`.
+
+**`novelty_score`** — a float between 0.0 and 1.0 indicating how novel or surprising the information is. Routine performance reports score low; unexpected regulatory actions score high. This value feeds into the novelty bonus component of the signal weighting formula described in [Page 3](03-signal-scoring-and-weighted-signals.md).
+
+**`confidence`** — a float between 0.0 and 1.0 representing the model's confidence in the accuracy of its extraction. Lower values indicate ambiguous or incomplete source text. This value becomes the confidence gate input for signal scoring.
+
+**`extraction_warnings`** — a list of issues encountered during extraction, such as `ambiguous_ticker`, `incomplete_text`, or `low_confidence`. These warnings are persisted alongside the intelligence record for operational monitoring.
+
+The JSON schema is generated programmatically from the Pydantic models via `generate_json_schema()` in `services/extractor/schemas.py`, which calls Pydantic's `model_json_schema()` and then inlines all `$defs` references so the schema is self-contained and Ollama-friendly.
+
+---
+
+## The Global Event Classifier
+
+Not all documents describe entity-specific developments. Macro news articles — those tagged with `document_type='macro_event'` by the Parser — describe events that affect entire sectors or economies: trade disputes, central bank rate decisions, commodity supply disruptions, geopolitical conflicts. These documents are routed to the `app:queue:macro_classification` Redis queue and processed by the Global Event Classifier agent, registered under the slug `event-classifier` in the `ai_agents` table.
+
+The classifier is implemented in `services/extractor/event_classifier.py`. When the extractor worker in `services/extractor/main.py` pops a job and determines that the document type is `macro_event` (either because the job came from the macro queue or because the `documents` table records it as such), it routes the document to `_process_macro_classification()` instead of the standard extraction pipeline. This function calls `classify_global_event()`, which builds a dedicated prompt, sends it to Ollama through the same `OllamaClient` infrastructure, parses the response, and persists the result.
+
+The classifier's system prompt is distinct from the extractor's. It establishes the model's role as a macro-level news classifier and includes explicit anti-hallucination rules that are critical to preventing the classifier from overreaching. The prompt states that the model should only classify articles about macro events that affect entire sectors or economies — trade disputes, interest rate changes, commodity supply disruptions, regulatory changes, geopolitical conflicts, natural disasters. It explicitly lists what should not be classified as macro events: individual entity performance reports, lawsuits against a single entity, single-entity management changes, individual entity analysis, entity-specific debt or bankruptcy, and product launches by one entity. For these entity-specific articles that were incorrectly routed, the model is instructed to set severity to `"low"`, confidence below 0.3, and leave the `affected_regions`, `affected_sectors`, and `affected_commodities` arrays empty.
+
+The user prompt, built by `build_event_classification_prompt()`, reinforces these anti-hallucination rules and provides additional guidance. It instructs the model to only extract facts explicitly stated in the text, to set confidence below 0.4 for vague or speculative content, to distinguish announced policy from rumored policy, and to reserve `"critical"` severity for events affecting multiple countries or entire global systems. Articles longer than 6,000 characters are truncated before inclusion in the prompt.
+
+The output schema is the `GlobalEvent` dataclass, which contains:
+
+- `event_types` — a list of impact type strings, drawn from a fixed set: `supply_disruption`, `demand_shift`, `cost_increase`, `regulatory_pressure`, `currency_impact`, `commodity_shock`, `trade_barrier`, and `geopolitical_risk`. The model is instructed to include all applicable types rather than collapsing to a single category.
+- `severity` — one of `low`, `moderate`, `high`, or `critical`.
+- `affected_regions` — ISO 3166-1 alpha-2 country codes or region names (e.g., `US`, `CN`, `EU`, `GB`, `JP`). Only regions explicitly mentioned or clearly implied should be included.
+- `affected_sectors` — GICS sector identifiers such as `Energy`, `Financials`, `Information Technology`, or `Industrials`.
+- `affected_commodities` — commodity identifiers like `crude_oil`, `natural_gas`, `gold`, `copper`, `wheat`, `lithium`, or `semiconductors`. An empty list if no commodities are directly affected.
+- `summary` — a one-to-three sentence summary of the event and its domain implications.
+- `key_facts` — facts explicitly stated in the article, limited to three to five items.
+- `estimated_duration` — one of `short_term` (days to weeks), `medium_term` (weeks to months), or `long_term` (months to years).
+- `confidence` — a float between 0.0 and 1.0, clamped during parsing.
+
+Each `GlobalEvent` also carries a `model_metadata` object recording the provider (`ollama`), model name, prompt version (`event-classification-v1`), and schema version (`1.0.0`), plus a `source_document_id` linking back to the originating document.
+
+After a successful classification, the system computes macro impact records for all tracked entities using the exposure-based interpolation engine in `services/aggregation/interpolation.py`. Each entity's exposure profile — geographic revenue mix, supply chain regions, key input commodities, regulatory jurisdictions, and position tier — determines how much a given macro event affects that entity. Entities with non-zero macro impact scores get `macro_impact_records` rows persisted to PostgreSQL, and aggregation jobs are enqueued to `app:queue:aggregation` for each affected entity identifier. The extractor worker tracks consecutive macro classification failures and emits a critical-level alert after three consecutive failures, continuing with entity-only signals in the meantime.
+
+---
+
+## The JSON Repair Pipeline
+
+LLM output is inherently unreliable at the syntactic level. Models sometimes wrap JSON in markdown fences, produce trailing commas, leave strings unterminated, or truncate output mid-object when they hit token limits. The extractor addresses this with a three-stage JSON repair pipeline implemented across `services/extractor/client.py` and `services/extractor/schemas.py`.
+
+The first stage is a direct `json.loads()` call. If the raw model output is already valid JSON, no repair is needed and the pipeline moves straight to validation. This is the fast path for well-behaved model responses.
+
+The second stage strips markdown fences. Models frequently wrap their output in `` ```json ... ``` `` blocks despite being told not to. The `_strip_markdown_fences()` function in `services/extractor/client.py` uses a regex to detect and remove these wrappers before attempting another parse.
+
+The third stage invokes the `json-repair` library as a fallback. The `_repair_json()` function in `services/extractor/client.py` calls `repair_json()` with `return_objects=False` to get a repaired JSON string. This library handles a wide range of common LLM JSON errors — trailing commas, missing quotes, unescaped characters — that would otherwise require custom repair logic.
+
+The `services/extractor/schemas.py` module contains an additional layer of repair logic in its own `_repair_json()` function, which handles cases that the library might miss. It strips non-JSON prefixes (models sometimes prepend explanatory text before the opening brace), removes control characters that break parsing, fixes trailing commas before closing brackets, and as a last resort calls `_repair_truncated_json()` — a state-machine parser that walks the string tracking bracket depth and string state, then appends the necessary closing tokens to complete a truncated JSON object.
+
+For the Global Event Classifier, the `_parse_classification_response()` function in `services/extractor/event_classifier.py` reuses the same `_strip_markdown_fences()` and `_repair_json()` functions from the client module, and additionally handles the case where the model wraps the output object in a single-element list — a quirk observed with some model configurations.
+
+---
+
+## Structural and Semantic Validation
+
+Repairing JSON syntax is only the first step. The `validate_extraction()` function in `services/extractor/schemas.py` performs both structural and semantic validation on the parsed output, and the distinction between the two is important for understanding the retry logic.
+
+Structural validation begins with normalization. The `_normalize_extraction_data()` function fills in missing top-level fields with sensible defaults (empty summary, empty companies array, 0.5 novelty score, 0.3 confidence), clamps numeric fields to the [0.0, 1.0] range, and normalizes per-entity fields. Catalyst types that the model produces as free-text alternatives — `"strategic pivot"`, `"acquisition"`, `"lawsuit"`, `"inflation"`, `"launch"` — are mapped to their canonical enum values through a comprehensive alias dictionary. Impact horizons like `"long-term"`, `"short"`, `"immediate"`, or `"near-term"` are similarly mapped to the valid set (`intraday`, `1d`, `1d_7d`, `1d_30d`, `30d_90d`, `90d_plus`). After normalization, the data is validated against the `ExtractionResult` Pydantic model, which enforces type constraints, enum membership, and range bounds.
+
+Semantic validation catches issues that are structurally valid but logically suspect. The `_semantic_checks()` function runs a series of cross-field consistency checks that produce either errors (which trigger a retry) or warnings (which are logged but do not block acceptance). Semantic errors include duplicate entity identifiers across entity entries, missing identifier fields, and invalid impact horizon values. Semantic warnings include empty summaries, low confidence with entities present, invalid identifier formats (not matching the one-to-five uppercase letter pattern), missing evidence spans, evidence spans that are too short (under 8 characters) or too long (over 500 characters), high impact scores with no supporting key facts, very low relevance scores, and strong sentiment paired with negligible impact scores.
+
+When the original document text is available, the validator also performs an evidence grounding check: each evidence span is searched for in the source text (case-insensitive), and spans not found in the document are flagged with a warning. This helps detect hallucinated evidence — quotes the model fabricated rather than extracted from the actual text.
+
+If validation produces any semantic errors, the `ValidationReport` is marked as invalid and the `OllamaClient` retry loop treats it as a failed attempt. The retry logic uses exponential backoff with configurable parameters: a base delay (default from `OllamaConfig`), a multiplier applied on each retry, and a maximum delay cap. The number of retries is configurable per agent through the `max_retries` field in the `ai_agents` or `agent_variants` table. Non-retryable errors — HTTP 400, 401, 403, 404, and 422 responses from Ollama — short-circuit the retry loop immediately, since these indicate a problem with the request itself rather than a transient model failure.
+
+Every attempt, whether successful or not, is recorded in an `ExtractionAttempt` dataclass that captures the raw output, validation report, error description, duration in milliseconds, model name, and whether the error was retryable. The full list of attempts is preserved in the `ExtractionResponse` for audit purposes and uploaded to MinIO by the persistence layer.
+
+---
+
+## The AgentConfigResolver: Hot-Swapping Models and Prompts
+
+Both the Document Intelligence Extractor and the Global Event Classifier resolve their runtime configuration through the `AgentConfigResolver` in `services/shared/agent_config.py`. This mechanism allows operators to change models, prompts, timeouts, retry counts, and token budgets without restarting any service — changes take effect within 60 seconds.
+
+The resolver works by querying the `ai_agents` and `agent_variants` PostgreSQL tables with a single SQL statement that uses `COALESCE` to prefer variant values over base agent values. When the extractor worker starts, it creates an `AgentConfigResolver` instance with a 60-second TTL cache and calls `resolver.resolve("document-extractor")` to get the active configuration. If an active variant exists for the agent (enforced by a unique partial index on `agent_variants` that allows at most one active variant per agent), the variant's `model_name`, `system_prompt`, `temperature`, `max_tokens`, `context_window`, `timeout_seconds`, and `max_retries` override the base agent's values wherever the variant provides a non-NULL value. If no active variant exists, the base agent's configuration is used. If the database query fails entirely, the resolver returns `None` and the worker falls back to environment-variable-based `OllamaConfig` defaults.
+
+The resolved configuration is captured in a `ResolvedAgentConfig` frozen dataclass that includes the `agent_id`, `variant_id` (if any), `model_provider`, `model_name`, `system_prompt`, `user_prompt_template`, `prompt_version`, `temperature`, `max_tokens`, `context_window`, `input_token_limit`, `token_budget`, `timeout_seconds`, and `max_retries`. The extractor worker uses this to build an `OllamaConfig` that is passed to the `OllamaClient`.
+
+The 60-second TTL cache means the resolver only hits the database once per minute per agent slug. Cache entries are keyed by slug and timestamped with `time.monotonic()`. When a cached entry expires, the next `resolve()` call re-queries the database and refreshes the cache. The `invalidate()` method can clear a single slug or the entire cache, though in practice the TTL-based expiry is sufficient for normal operations.
+
+The extractor worker re-resolves its configuration every 100 jobs. If the resolved model name has changed (for example, because an operator activated a variant that uses a different model), the worker closes the old `OllamaClient` and creates a new one with the updated configuration. The event classifier is resolved separately and can use a different model than the document extractor — the worker maintains two independent `OllamaClient` instances when the models differ.
+
+Token budget enforcement adds another layer of control. If a variant specifies a `token_budget` (total tokens per hour), the worker checks the `agent_performance_log` table before each invocation to see whether the budget has been exceeded. If so, the invocation is skipped entirely. Input token limits work similarly: if a variant sets an `input_token_limit`, the worker truncates the document text to approximately that many tokens (estimated at four characters per token) before sending it to the model.
+
+For a complete guide to creating variants, activating them, and comparing their performance, see the [AI Agents Guide](../ai-agents.md).
+
+---
+
+## Persistence: From Extraction to Database
+
+Once the LLM produces a valid extraction and it passes validation, the `persist_extraction()` function in `services/extractor/worker.py` orchestrates the full persistence pipeline. This function writes to both MinIO (for audit) and PostgreSQL (for downstream consumption), ensuring that every extraction attempt is fully traceable.
+
+The MinIO persistence layer uploads four artifacts per extraction, all stored under date-partitioned paths in dedicated buckets. The prompt metadata (prompt version, schema version, model name) goes to `app-llm-prompts`. The raw model output for every attempt — including failed ones — goes to `app-llm-results`, preserving the full retry history. A validation report summarizing the final attempt's status, errors, and warnings is uploaded alongside the raw output. On success, the final parsed intelligence object (the `ExtractionResult` serialized as JSON) is uploaded to a separate path for easy retrieval.
+
+The PostgreSQL persistence writes to two tables. The `document_intelligence` table receives one row per document, containing the summary, macro themes, novelty score, source credibility, extraction warnings, confidence, model metadata (provider, model name, prompt version, schema version), references to the MinIO artifacts (raw output ref, prompt ref), validation status (`valid` or `failed`), validation errors, and retry count. This row is the authoritative record of what the AI extracted from the document.
+
+The `document_impact_records` table receives one row per entity mention within the extraction. Each impact record is linked to the parent `document_intelligence` row via `intelligence_id` and to the `companies` table via `company_id`. The record captures the entity identifier, relevance, sentiment, impact score, impact horizon, catalyst type, key facts, risks, and evidence spans for that specific entity. The `company_id` is resolved from an identifier-to-UUID mapping that the worker maintains by querying the `companies` table (refreshed every 100 jobs). If an identifier in the extraction output does not match any tracked entity, the impact record is skipped with a warning — the system only persists impact records for entities in its tracked universe.
+
+After persisting the intelligence and impact records, the worker updates the document's status in the `documents` table to `extracted` (or `extraction_failed` if all retry attempts were exhausted). Even failed extractions get a `document_intelligence` row with `validation_status='failed'`, empty summary, zero confidence, and the accumulated error messages — this ensures the failure is visible in the database rather than silently lost.
+
+Performance metrics are collected for every extraction via `collect_metrics()` in `services/extractor/metrics.py` and persisted to a metrics table. Prometheus counters and histograms track extraction attempts, duration, retries, confidence distribution, validation errors, and estimated token usage (input and output, estimated at four characters per token). When a resolved agent config is available, the worker also logs to the `agent_performance_log` table with variant attribution, enabling the A/B comparison queries described in the [AI Agents Guide](../ai-agents.md).
+
+For the Global Event Classifier, persistence follows a parallel path. The prompt and raw output are uploaded to MinIO under an `event_classification/macro/` path prefix. The parsed `GlobalEvent` is persisted to the `global_events` PostgreSQL table, which stores the event types, severity, affected regions, affected sectors, affected commodities, summary, key facts, estimated duration, confidence, source document ID, and model metadata. Downstream, the macro interpolation engine computes `macro_impact_records` for each affected entity and persists those as well.
+
+---
+
+## Enqueuing Aggregation Jobs
+
+The final step in the extraction pipeline is to notify the downstream aggregation engine that new intelligence is available. After a successful document extraction, the worker pushes a job onto the `app:queue:aggregation` Redis list containing the identifier of the affected entity. The aggregation engine (described in [Page 3](03-signal-scoring-and-weighted-signals.md)) will pick up this job and recompute the weighted signals and trend summaries for that entity, incorporating the freshly extracted intelligence.
+
+For macro events, the enqueue logic is more expansive. After the Global Event Classifier produces a `GlobalEvent` and the interpolation engine computes macro impact records, the worker enqueues an aggregation job for every entity identifier that received a non-zero macro impact score. A single macro event — say, a new regulatory policy change affecting the Energy and Industrials sectors — can trigger aggregation recomputation for dozens of entities simultaneously. The aggregation job payload includes both the entity identifier and the `macro_event_id`, so the aggregation engine knows to incorporate the new macro signals.
+
+The worker alternates between the extraction and macro classification queues to prevent starvation: every third job is pulled from `app:queue:macro_classification`, with the remaining two-thirds from `app:queue:extraction`. If the preferred queue is empty, the worker falls back to the other queue, ensuring that neither pipeline stalls while the other has work available.
+
+---
+
+## What Comes Next
+
+At this point, documents have been transformed from unstructured text into structured JSON intelligence — `ExtractionResult` objects for entity-specific documents and `GlobalEvent` objects for macro news. These structured records are persisted in PostgreSQL and their entity identifiers have been enqueued for aggregation. But raw extraction output is not yet actionable for downstream decisions. The extraction tells us that a document is negative for Entity-A with an impact score of 0.7 and a confidence of 0.8, but it does not tell us how much weight that signal should carry relative to other signals about Entity-A, or how it compares to signals from different sources, time periods, or environmental conditions. [Page 3 — Signal Scoring and the WeightedSignal Abstraction](03-signal-scoring-and-weighted-signals.md) picks up the story from here, explaining how the aggregation engine transforms these raw extraction outputs into weighted signals through confidence gating, recency decay, source credibility scoring, novelty bonuses, and environmental context multipliers.
diff --git a/docs/sanitized-pipeline-deep-dive/03-signal-scoring-and-weighted-signals.md b/docs/sanitized-pipeline-deep-dive/03-signal-scoring-and-weighted-signals.md
new file mode 100644
index 0000000..d71a14c
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/03-signal-scoring-and-weighted-signals.md
@@ -0,0 +1,210 @@
+# Page 3 — Signal Scoring and the WeightedSignal Abstraction
+
+The extraction pipeline described in [Page 2](02-ai-agent-processing-and-extraction.md) produces structured intelligence records — `document_impact_records` for entity-specific documents, `macro_impact_records` for global events, and `competitive_signal_records` for cross-entity pattern propagation. Each record carries a sentiment, an impact score, a confidence value, and a publication timestamp. But these raw values are not directly comparable. A high-confidence extraction from a reputable source published ten minutes ago should carry far more weight than a low-confidence extraction from an unknown source published three weeks ago. A document that breaks genuinely novel information should matter more than one that rehashes yesterday's performance report. And when conditions are changing fast — high volatility, surging volume — fresh signals become even more critical.
+
+The signal scoring layer in `services/aggregation/scoring.py` solves this problem by transforming each raw intelligence record into a `WeightedSignal` object: a document reference paired with a composite aggregation weight that encodes recency, credibility, novelty, confidence, and environmental conditions into a single number. This page explains how that weight is computed, how sentiment labels become numeric values, and how three independent signal layers — Entity-Specific, Macro, and Competitive — each produce `WeightedSignal` objects that are concatenated into a unified list before the aggregation engine computes trend summaries. For a visual breakdown of the composite weight formula, see the [Weighted Signal Computation diagram](diagrams/weighted-signal-computation.md). For the full picture of how the three layers merge, see the [Three-Layer Signal Merging diagram](diagrams/three-layer-signal-merging.md).
+
+---
+
+## The WeightedSignal and SignalWeight Dataclasses
+
+The core abstraction is the `WeightedSignal` dataclass, defined in `services/aggregation/scoring.py`. It pairs a document reference with the computed weight and the signal's sentiment and impact values:
+
+- **`document_id`** — the UUID of the source document (for entity-specific and macro signals) or a synthetic identifier for pattern-derived signals (e.g., `pattern:Entity-A:performance_report:7d`).
+- **`weight`** — a `SignalWeight` object containing the component breakdown and the final combined score.
+- **`sentiment_value`** — a numeric sentiment value: `+1.0` for positive, `-1.0` for negative, `0.0` for neutral or mixed.
+- **`impact_score`** — the magnitude of impact, drawn from the extraction's per-entity impact score for entity-specific signals, or scaled by a layer-specific weight multiplier for macro and competitive signals.
+
+The `SignalWeight` dataclass captures the individual components that feed into the combined weight, making the scoring decision fully transparent and auditable:
+
+- **`recency`** — the exponential decay weight based on document age.
+- **`credibility`** — the source credibility weight after clamping and exponentiation.
+- **`novelty_bonus`** — the additive bonus derived from the document's novelty score.
+- **`confidence_gate`** — either `1.0` (signal passes) or `0.0` (signal is gated out).
+- **`market_ctx_multiplier`** — a multiplicative boost from environmental conditions, always `>= 1.0`.
+- **`combined`** — the final composite weight used by the aggregation engine.
+
+The `ScoringConfig` frozen dataclass holds all tunable parameters for the scoring functions — half-life hours per window, credibility bounds, novelty bonus cap, confidence floor, and environmental context thresholds. A module-level `DEFAULT_CONFIG` singleton provides the production defaults, but every scoring function accepts an optional `config` parameter so that tests and alternative configurations can override any parameter without modifying global state.
+
+---
+
+## The Composite Weight Formula
+
+The `compute_signal_weight()` function in `services/aggregation/scoring.py` computes the combined weight for a single document signal. The formula is:
+
+```
+combined = gate × recency × credibility × (1 + novelty_bonus) × market_context_multiplier
+```
+
+Each factor is computed independently and then multiplied together. This multiplicative structure means that any single factor can zero out the entire weight (the confidence gate) or amplify it (the market context multiplier), and the interaction between factors is naturally captured — a highly credible, very recent document with novel information in a volatile environment receives the maximum possible weight, while a stale, low-credibility document with routine information receives a weight close to zero.
+
+The following sections describe each component in detail.
+
+---
+
+## Confidence Gate
+
+The confidence gate is the first and most decisive filter. If the extraction confidence for a document falls below the `confidence_floor` threshold — set to `0.2` in the default `ScoringConfig` — the gate evaluates to `0.0` and the entire combined weight becomes zero. The document is effectively excluded from aggregation. If the confidence meets or exceeds the threshold, the gate evaluates to `1.0` and has no further effect on the weight.
+
+This binary gate exists because documents with very low extraction confidence are too unreliable to aggregate. A confidence of 0.15 typically means the LLM struggled to parse the document — perhaps the text was truncated, the language was ambiguous, or the document type was unusual. Including such signals would add noise rather than information. The threshold of 0.2 is deliberately low; it filters only the most unreliable extractions while allowing moderately confident signals to participate (their lower confidence is reflected through the credibility component instead).
+
+---
+
+## Recency Decay
+
+The `recency_weight()` function computes an exponential decay based on how old a document is relative to the aggregation anchor time. The formula is:
+
+```
+w = 2^(−age_hours / half_life)
+```
+
+A document published exactly one half-life ago receives a recency weight of `0.5`. A document published two half-lives ago receives `0.25`, and so on. A document published at or after the reference time receives the maximum weight of `1.0`.
+
+The half-life varies by trend window, reflecting the intuition that shorter windows need faster decay to stay responsive, while longer windows should give older documents more influence. The default half-lives, configured in `ScoringConfig.half_life_hours`, are:
+
+| Window | Half-Life |
+|--------|-----------|
+| `intraday` | 2 hours |
+| `1d` | 12 hours |
+| `7d` | 72 hours (3 days) |
+| `30d` | 240 hours (10 days) |
+| `90d` | 720 hours (30 days) |
+
+For the intraday window, a document published four hours ago already has a recency weight of `0.25` — it is rapidly losing influence as newer information arrives. For the 90-day window, that same four-hour-old document still has a recency weight of essentially `1.0`, because the 30-day half-life means age only becomes significant over weeks.
+
+A floor value of `min_recency_weight = 0.01` prevents very old documents from being completely zeroed out. Even a document from months ago retains a trace-level weight of 1%, ensuring it can still contribute to trend computation if no newer signals exist. Both timestamps are normalized to UTC; naive datetimes are treated as UTC to avoid timezone-related scoring errors.
+
+---
+
+## Source Credibility
+
+The `credibility_weight()` function transforms a source's credibility score into a weight component. The raw credibility value — a float between 0.0 and 1.0 stored in the `document_intelligence` table — is first clamped to the range `[0.1, 1.0]` using the `credibility_floor` and `credibility_ceiling` parameters from `ScoringConfig`. This clamping ensures that even the least credible sources retain a minimum weight of 0.1 rather than being completely silenced, while preventing any source from exceeding a weight of 1.0.
+
+After clamping, the value is raised to the `credibility_exponent` power. The default exponent is `1.0`, which means the clamped credibility passes through unchanged. Setting the exponent above 1.0 would penalize low-credibility sources more aggressively — for example, an exponent of 2.0 would reduce a credibility of 0.5 to a weight of 0.25. Setting it below 1.0 would flatten the curve, making the system more tolerant of lower-credibility sources. The exponent is configurable through `ScoringConfig` to allow operators to tune the credibility sensitivity without changing the scoring code.
+
+---
+
+## Novelty Bonus
+
+The novelty bonus rewards documents that contain genuinely new information. The bonus is computed as:
+
+```
+novelty_bonus = novelty_score × novelty_bonus_max
+```
+
+where `novelty_score` is the 0.0-to-1.0 value produced by the extraction model (see the `ExtractionResult` schema in [Page 2](02-ai-agent-processing-and-extraction.md)) and `novelty_bonus_max` is `0.25` by default. This means the bonus ranges from `0.0` (completely routine information) to `0.25` (maximally novel information), providing up to a 25% boost to the signal weight.
+
+The bonus enters the composite formula as `(1 + novelty_bonus)`, so it acts as a multiplicative amplifier on the base weight. A document with a novelty score of 1.0 gets its weight multiplied by 1.25; a document with a novelty score of 0.0 gets multiplied by 1.0 (no change). This design ensures that novelty can only increase a signal's weight, never decrease it — routine information is not penalized, it simply does not receive the bonus.
+
+---
+
+## Environmental Context Multiplier
+
+The `market_context_multiplier()` function computes a boost factor based on real-time environmental conditions for the entity being aggregated. The multiplier is always `>= 1.0`, meaning environmental context can only amplify signal weights, never reduce them. When no environmental context data is available (the `MarketContext` object from `services/shared/schemas.py` has `has_data == False`), the multiplier defaults to `1.0`.
+
+Two environmental features contribute to the boost:
+
+**Volatility boost.** When the entity's price volatility exceeds the `volatility_recency_boost_threshold` (default `1.0` in price units), the excess volatility is transformed through a logarithmic scaling function: `log₁₊(excess) × 0.15`. The logarithmic scaling prevents extreme volatility from producing runaway weight amplification. The boost is capped at `volatility_recency_boost_max = 0.30`, so the maximum volatility contribution is a 30% weight increase. The rationale is that in highly volatile environments, fresh intelligence is disproportionately valuable — a signal about Entity-C matters more when Entity-C is swinging 5% intraday than when it is moving in a tight range.
+
+**Volume surge boost.** When the entity's volume change percentage exceeds `volume_surge_threshold_pct = 50.0%` (meaning activity volume is at least 50% above the prior period's average), a flat `volume_surge_boost = 0.15` is added. Unlike the volatility boost, this is binary — either the volume threshold is met and the full 15% boost applies, or it is not and no boost is added. High-volume moves carry more conviction because they represent broader participation rather than thin-activity noise.
+
+The two boosts are additive within the multiplier: `multiplier = 1.0 + volatility_boost + volume_surge_boost`. In the most extreme case — high volatility and a volume surge — the combined multiplier reaches `1.0 + 0.30 + 0.15 = 1.45`, amplifying the signal weight by 45%. The `MarketContext` data is fetched by `services/aggregation/market_context.py` from the data tables in PostgreSQL, using the same entity identifier and window parameters as the impact record query.
+
+---
+
+## Sentiment Mapping
+
+Before signals can be aggregated into trend summaries, the categorical sentiment labels from the extraction output must be converted to numeric values. The `sentiment_to_numeric()` function in `services/aggregation/scoring.py` performs this mapping:
+
+| Sentiment Label | Numeric Value |
+|----------------|---------------|
+| `positive` | `+1.0` |
+| `negative` | `-1.0` |
+| `neutral` | `0.0` |
+| `mixed` | `0.0` |
+
+The mapping is case-insensitive. Any unrecognized label defaults to `0.0`. The choice to map both `neutral` and `mixed` to `0.0` is deliberate — a mixed-sentiment document (one that contains both positive and negative signals for the same entity) should not push the trend in either direction. The contradiction between the positive and negative aspects is captured separately by the contradiction detection system described in [Page 4](04-trend-aggregation-and-accumulating-signals.md), rather than being baked into the sentiment value itself.
+
+For macro signals, the direction-to-sentiment mapping in `services/aggregation/worker.py` follows the same pattern: `positive` maps to `+1.0`, `negative` to `-1.0`, and both `mixed` and `neutral` to `0.0`. For competitive signals built by `build_pattern_weighted_signals()` in `services/aggregation/signal_propagation.py`, the sentiment is derived from the pattern's directional bias: `+1.0` if `positive_pct > negative_pct`, `-1.0` otherwise.
+
+---
+
+## Weighted Sentiment Average
+
+The `weighted_sentiment_average()` function computes the central metric that drives trend direction: a weight-adjusted average sentiment across all signals for an entity in a given window. The formula is:
+
+```
+weighted_avg = Σ(combined_weight × impact_score × sentiment_value) / Σ(combined_weight × impact_score)
+```
+
+Each signal contributes its sentiment value scaled by both its composite weight and its impact score. The denominator normalizes by the total effective weight, producing a value in the range `[-1.0, +1.0]`. A result near `+1.0` means the weighted evidence is overwhelmingly positive; near `-1.0` means overwhelmingly negative; near `0.0` means either neutral or evenly split.
+
+The use of `combined_weight × impact_score` as the effective weight means that high-impact, high-weight signals dominate the average. A single high-confidence, recent, credible document with a strong impact score can outweigh several older, lower-impact documents — which is the intended behavior. The aggregation engine in `services/aggregation/worker.py` passes this weighted average to `derive_trend_direction()`, which maps it to a `TrendDirection` enum value (positive, negative, mixed, or neutral) using the thresholds described in [Page 4](04-trend-aggregation-and-accumulating-signals.md).
+
+If the total effective weight is zero — either because no signals exist or all signals were gated out by the confidence floor — the function returns `0.0`, which maps to a neutral trend direction.
+
+---
+
+## The Three Signal Layers
+
+The aggregation engine in `services/aggregation/worker.py` does not treat all intelligence sources equally. Signals flow through three independent layers, each with a different relative weight, before being concatenated into a single `WeightedSignal` list for trend computation. This layered architecture allows the system to incorporate diverse intelligence sources while controlling how much influence each source type has on the final trend.
+
+### Layer 1 — Entity-Specific Signals (Weight: 1.0)
+
+Entity-specific signals are the primary layer. They are built by `build_weighted_signals()` in `services/aggregation/worker.py` from `document_impact_records` — the per-entity extraction output produced by the Document Intelligence Extractor (see [Page 2](02-ai-agent-processing-and-extraction.md)). Each impact record's sentiment is converted via `sentiment_to_numeric()`, and its impact score is used directly without any layer-level scaling. The `compute_signal_weight()` function produces the composite weight using the document's publication time, source credibility, novelty score, extraction confidence, and the entity's current environmental context.
+
+Entity-specific signals carry a relative weight of `1.0` — they are the baseline against which other layers are measured. This reflects the design principle that direct, entity-specific intelligence (a performance report about Entity-A, a product launch by Entity-B, a lawsuit against Entity-E) is the most relevant and reliable signal for that entity's trend.
+
+### Layer 2 — Macro Signals (Weight: 0.3)
+
+Macro signals capture the indirect impact of global events on individual entities. They are built by `build_macro_weighted_signals()` in `services/aggregation/worker.py` from `macro_impact_records` — the per-entity impact scores computed by the exposure-based interpolation engine after the Global Event Classifier processes a macro news article. The sentiment is mapped from the `impact_direction` field (`positive` → `+1.0`, `negative` → `-1.0`, `mixed`/`neutral` → `0.0`), and the impact score is scaled by `MACRO_SIGNAL_WEIGHT`, which defaults to `0.3` in `AggregationConfig`.
+
+The 0.3 weight means that a macro signal's impact score is reduced to 30% of its raw value before entering the aggregation. This attenuation reflects the inherent uncertainty in macro-to-entity impact estimation — a policy change might affect Entity-D's revenue, but the magnitude depends on exposure profiles, supply chain flexibility, and competitive dynamics that the interpolation engine can only approximate. By weighting macro signals at 0.3 relative to entity-specific signals at 1.0, the system ensures that macro intelligence informs the trend without overwhelming direct entity-specific evidence.
+
+The recency decay, credibility, and confidence gating for macro signals use the same `compute_signal_weight()` function as entity-specific signals. The `published_at` timestamp comes from the global event's source document (the macro news article), and the `source_credibility` and `extraction_confidence` both use the macro impact record's `confidence` field.
+
+### Layer 3 — Competitive Signals (Weight: 0.2)
+
+Competitive signals capture cross-entity effects: when a catalyst hits one entity, historical patterns suggest how competitors might be affected. They are built by `build_pattern_weighted_signals()` in `services/aggregation/signal_propagation.py` from two sources: `HistoricalPattern` objects (self-entity patterns mined by `services/aggregation/pattern_matcher.py`) and `CompetitiveSignalRecord` objects (cross-entity propagation signals stored in `competitive_signal_records`).
+
+For historical patterns, the sentiment is derived from the pattern's directional bias (`+1.0` if `positive_pct > negative_pct`, `-1.0` otherwise), and the impact score is the pattern's `avg_strength` multiplied by `competitive_signal_weight` (default `0.2` from `CompetitiveConfig`). The `published_at` for recency decay uses the pattern's `data_end` — the most recent data point in the pattern's sample — and the `extraction_confidence` uses the pattern's `pattern_confidence`. Source credibility is set to `1.0` because patterns are derived from validated historical data, and novelty is fixed at `0.5`.
+
+For competitive signal records, the same structure applies: sentiment from `signal_direction`, impact from `signal_strength × competitive_signal_weight`, recency from `computed_at`, and confidence from `pattern_confidence`.
+
+The 0.2 weight makes competitive signals the lightest layer. This is appropriate because competitive signal propagation involves the most inference — the system is predicting how Entity B will react based on what happened to Entity A in historically similar situations. The signal is valuable as supplementary evidence but should not drive trend direction on its own.
+
+---
+
+## Signal Merging in the Aggregation Engine
+
+The `aggregate_company_window()` function in `services/aggregation/worker.py` orchestrates the merging of all three layers for a single entity and window. The process follows a clear sequence:
+
+1. **Fetch entity-specific impact records** from `document_impact_records` for the entity within the window's time range.
+2. **Fetch environmental context** for the entity from data tables.
+3. **Build entity-specific weighted signals** via `build_weighted_signals()`.
+4. **Check the macro toggle** — query `risk_configs` for the `macro_enabled` flag, then fetch and merge macro signals if enabled.
+5. **Check the competitive toggle** — query `risk_configs` for the `competitive_enabled` flag, then fetch patterns, fetch competitive signals, and merge if enabled.
+6. **Concatenate** all `WeightedSignal` lists into a single list.
+7. **Assemble the `TrendSummary`** from the merged signals.
+
+The concatenation in step 6 is a simple list append — `signals = signals + macro_signals` followed by `signals = signals + pattern_weighted`. There is no re-weighting or normalization at the merge point. The relative influence of each layer is already encoded in the impact scores (scaled by 0.3 for macro, 0.2 for competitive, 1.0 for entity-specific) and in the composite weights computed by `compute_signal_weight()`. The `weighted_sentiment_average()` function then naturally produces a sentiment average that reflects these relative weights.
+
+---
+
+## Runtime Toggles and Graceful Degradation
+
+Both the macro and competitive signal layers can be enabled or disabled at runtime through the `risk_configs` PostgreSQL table, without restarting any service. The toggle state is read fresh from the database at the start of every aggregation cycle — there is no caching — so changes take effect on the very next cycle.
+
+The `fetch_macro_enabled()` function in `services/aggregation/worker.py` queries the most recent active `risk_configs` row and reads the `config->>'macro_enabled'` JSON field. If the field is explicitly set to `"true"` or `"false"`, that value overrides the `AggregationConfig` default. If no config row exists or the field is absent, the function returns `None` and the engine falls back to the `AggregationConfig.macro_enabled` default (which is `True`). The `fetch_competitive_enabled()` function follows the identical pattern for the `competitive_enabled` field.
+
+When a layer is disabled, the aggregation engine simply skips the fetch-and-merge step for that layer. Entity-specific signals are always computed — they cannot be toggled off. This means the system degrades gracefully: disabling the macro layer produces trends based on entity-specific signals alone (plus competitive signals if enabled), and disabling the competitive layer produces trends based on entity-specific and macro signals. Disabling both layers reduces the engine to its original single-layer behavior, using only direct document intelligence.
+
+Crucially, disabling a layer does not stop upstream processing. When the macro layer is disabled, the Global Event Classifier continues to classify macro events and the interpolation engine continues to compute `macro_impact_records`. The data accumulates in PostgreSQL. When the layer is re-enabled, the aggregation engine immediately picks up all the macro impact records that were computed while the layer was disabled — there is no data loss or gap in coverage. The same applies to competitive signals: pattern mining and signal propagation continue regardless of the toggle state.
+
+If the competitive signal fetch fails at runtime (for example, due to a database timeout), the aggregation engine catches the exception, logs it, and continues with entity-specific and macro signals only. This exception-based graceful degradation ensures that a transient failure in one layer does not block trend computation entirely.
+
+---
+
+## What Comes Next
+
+At this point, every document intelligence record, macro impact record, and competitive signal record has been transformed into a `WeightedSignal` with a composite weight that encodes recency, credibility, novelty, confidence, and environmental conditions. The three signal layers have been merged into a single list, and the weighted sentiment average has been computed. But a single aggregation cycle produces only a snapshot — a point-in-time view of the evidence. The real power of the system emerges when these snapshots accumulate across multiple documents and time windows, building a case for action. [Page 4 — Trend Aggregation and Accumulating Signals](04-trend-aggregation-and-accumulating-signals.md) explains how the aggregation engine computes `TrendSummary` objects across five time windows, how consecutive same-direction signals strengthen trend confidence and escalate the system's response from neutral observation to actionable decision recommendations, and how contradiction detection and evidence ranking ensure that the trend reflects genuine consensus rather than noise.
diff --git a/docs/sanitized-pipeline-deep-dive/04-trend-aggregation-and-accumulating-signals.md b/docs/sanitized-pipeline-deep-dive/04-trend-aggregation-and-accumulating-signals.md
new file mode 100644
index 0000000..bd417a1
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/04-trend-aggregation-and-accumulating-signals.md
@@ -0,0 +1,267 @@
+# Page 4 — Trend Aggregation and Accumulating Signals
+
+The scoring layer described in [Page 3](03-signal-scoring-and-weighted-signals.md) transforms every intelligence record into a `WeightedSignal` — a document reference paired with a composite weight that encodes recency, credibility, novelty, confidence, and environmental conditions. Three independent signal layers (Entity-Specific at weight 1.0, Environmental at 0.3, Relational at 0.2) each produce `WeightedSignal` objects that are concatenated into a single list. But a single list of weighted signals is still just raw material. The aggregation engine in `services/aggregation/worker.py` is where that raw material becomes a decision-grade assessment: a `TrendSummary` object that captures the direction, strength, confidence, contradiction level, and supporting evidence for an entity across a specific time window. This page explains how that transformation works — from weighted sentiment averages through trend direction derivation, contradiction detection, evidence ranking, and confidence computation — and, critically, how consecutive signals pointing in the same direction accumulate across documents and time windows to escalate the system's response from passive observation to actionable decision recommendations.
+
+For a visual overview of the accumulation and escalation process, see the [Trend Accumulation and Escalation diagram](diagrams/trend-accumulation-escalation.md). For how the three signal layers merge into the aggregation engine, see the [Three-Layer Signal Merging diagram](diagrams/three-layer-signal-merging.md).
+
+---
+
+## Five Time Windows
+
+The aggregation engine does not compute a single trend for each entity. It computes five, one for each time window defined in `services/aggregation/worker.py`:
+
+| Window | Lookback Duration |
+|--------|-------------------|
+| `intraday` | 12 hours |
+| `1d` | 1 day |
+| `7d` | 7 days |
+| `30d` | 30 days |
+| `90d` | 90 days |
+
+Each window produces an independent `TrendSummary` by fetching all impact records, macro impacts, and competitive signals for the entity within that window's time range. The `aggregate_company_window()` function in `services/aggregation/worker.py` orchestrates this per-window computation: it determines the time range from the window's lookback duration, fetches `document_impact_records` from PostgreSQL, retrieves environmental context, builds entity-specific weighted signals, checks the macro and competitive runtime toggles (see [Page 3](03-signal-scoring-and-weighted-signals.md) for toggle details), merges any enabled layer signals, and then assembles the `TrendSummary`.
+
+The five-window design serves a specific purpose. Short windows (intraday, 1d) capture fast-moving sentiment shifts — a breaking negative performance disclosure, a sudden regulatory action — while long windows (30d, 90d) reveal sustained trends that persist across many documents and data cycles. An entity might show a negative intraday trend after a single unfavorable article, but a neutral 30-day trend because the broader evidence base is balanced. The recommendation engine downstream (described in [Page 5](05-recommendation-generation.md)) evaluates each window's `TrendSummary` independently, so the system can respond to both short-term catalysts and long-term directional shifts.
+
+The `aggregate_company()` function iterates over all effective windows (configurable via `AggregationConfig.windows`, defaulting to all five) and calls `aggregate_company_window()` for each one. This means a single aggregation cycle for one entity produces up to five `TrendSummary` objects, each reflecting a different temporal perspective on the same underlying evidence.
+
+---
+
+## Trend Direction Derivation
+
+Once the weighted sentiment average has been computed from the merged signal list (see the `weighted_sentiment_average()` function described in [Page 3](03-signal-scoring-and-weighted-signals.md)), the `derive_trend_direction()` function in `services/aggregation/worker.py` maps that numeric value to a `TrendDirection` enum. The rules are evaluated in a specific order, and the first matching rule wins:
+
+1. **Mixed** — If the contradiction score exceeds `0.10` (the `MIXED_THRESHOLD` constant) *and* the absolute value of the average sentiment is below `0.30`, the direction is `MIXED`. This rule fires first because high contradiction with a weak directional signal indicates genuine disagreement in the evidence — the trend is not simply neutral, it is actively contested.
+
+2. **Positive** — If the average sentiment is `≥ 0.15` (the `POSITIVE_THRESHOLD` constant), the direction is `POSITIVE`. This means the weight-adjusted evidence leans favorable with enough conviction to cross the threshold.
+
+3. **Negative** — If the average sentiment is `≤ -0.15` (the `NEGATIVE_THRESHOLD` constant), the direction is `NEGATIVE`. The symmetric threshold ensures that positive and negative classifications require the same magnitude of evidence.
+
+4. **Neutral** — If none of the above conditions are met, the direction is `NEUTRAL`. This covers the range where the average sentiment falls between -0.15 and +0.15 without high contradiction — the evidence is either balanced or insufficient to establish a directional lean.
+
+The mixed-first evaluation order is important. Consider a scenario where five documents are positive and four are negative, all with similar weights. The weighted sentiment average might be slightly positive (say, +0.08), which would normally map to neutral. But the contradiction score — computed from the minority/majority weight split — would be high (close to 0.44). The mixed rule catches this case: the evidence is not neutral, it is conflicted. This distinction matters downstream because mixed trends receive different treatment in the recommendation engine than neutral trends.
+
+---
+
+## Contradiction Detection
+
+The contradiction detection module in `services/aggregation/contradiction.py` provides a structured analysis of disagreement within the signal set. Rather than collapsing contradictory evidence into a single number, it produces a `ContradictionResult` containing both an overall score and a list of `DisagreementDetail` objects that explain *where* the disagreement lies.
+
+The `detect_contradictions()` function runs two analyses:
+
+### Sentiment Disagreement
+
+The `_detect_sentiment_disagreement()` function examines whether both positive and negative sentiment signals exist in the signal set. For each signal with a non-zero effective weight (`combined_weight × impact_score > 0`), it classifies the signal as positive or negative based on its `sentiment_value` and accumulates the effective weight for each side. If both sides have at least one signal, it produces a `DisagreementDetail` with dimension `"sentiment"`, listing the document IDs and weights for each side, along with a human-readable description like "Sentiment split: 3 positive vs 2 negative signals (minority weight ratio 38%)".
+
+### Catalyst-Level Disagreement
+
+The `_detect_catalyst_disagreement()` function goes deeper. It groups signals by their `catalyst_type` (performance_report, product_launch, regulatory, etc.) using `CatalystEntry` objects built from the `document_impact_records`. Within each catalyst group, it checks whether both positive and negative signals exist. If they do, it produces a `DisagreementDetail` with dimension `"catalyst:<type>"` — for example, `"catalyst:performance_report"` when some documents interpret a periodic disclosure positively and others negatively. This catalyst-level analysis is valuable because it pinpoints the specific topic of disagreement rather than just flagging that disagreement exists somewhere in the evidence.
+
+### The Overall Contradiction Score
+
+The `_compute_overall_score()` function computes the backward-compatible scalar contradiction score using the minority/majority weight ratio formula:
+
+```
+contradiction_score = minority_weight / total_weight
+```
+
+where `minority_weight` is the smaller of the positive and negative effective weights, and `total_weight` is their sum. Signals with zero effective weight or neutral sentiment are excluded. The score ranges from `0.0` (complete agreement — all signals point the same direction) to `0.5` (perfect split — positive and negative weights are exactly equal). A score of `0.0` means no contradiction at all. A score above `0.10` combined with a weak average sentiment triggers the mixed direction classification in `derive_trend_direction()`.
+
+The contradiction score also feeds directly into the confidence computation as a penalty, described in the next section. High contradiction reduces the system's confidence in the trend, which in turn affects whether the trend can escalate to actionable recommendations.
+
+---
+
+## Evidence Ranking
+
+Not all documents contributing to a trend are equally important. The `rank_evidence()` function in `services/aggregation/worker.py` delegates to the evidence ranking module (`services/aggregation/evidence.py`) to produce ordered lists of the most influential supporting and opposing documents. The ranking uses a composite scoring approach configured by `EvidenceRankConfig`, considering multiple factors:
+
+- **Weight** — the signal's composite weight from the scoring layer, reflecting recency, credibility, novelty, confidence, and environmental context.
+- **Impact** — the extraction's impact score for the entity, reflecting how significant the document's content is.
+- **Recency** — how recently the document was published, with more recent documents ranked higher.
+- **Confidence** — the extraction confidence, reflecting how reliably the LLM parsed the document.
+
+Signals are split into supporting (positive sentiment) and opposing (negative sentiment) groups. Neutral and mixed sentiment signals are excluded from evidence lists — they do not argue for or against the trend direction. Within each group, signals are sorted by their composite rank score in descending order, and the top entries (up to `MAX_EVIDENCE_REFS = 10` per side) are returned as document ID lists.
+
+The `assemble_trend_with_evidence()` function in `services/aggregation/worker.py` uses the detailed variant `rank_evidence_detailed()` to get `RankedEvidence` objects that include the individual scoring components (weight, impact, recency, confidence, sentiment value). These detailed rankings are persisted to the `trend_evidence` table for auditability, while the document ID lists are stored directly in the `TrendSummary` as `top_supporting_evidence` and `top_opposing_evidence`.
+
+The evidence ranking serves two purposes. First, it provides the recommendation engine with the most relevant documents to cite in its thesis generation (see [Page 5](05-recommendation-generation.md)). Second, it gives human reviewers a quick way to understand *why* the system reached a particular trend assessment — the top-ranked documents are the ones that most influenced the direction and strength.
+
+---
+
+## Confidence Computation
+
+The `compute_trend_confidence()` function in `services/aggregation/worker.py` produces the confidence score for a `TrendSummary`. This score is critical because it directly gates whether a trend can produce actionable recommendations — the eligibility evaluation in `services/recommendation/eligibility.py` requires a minimum confidence of `0.35` to generate any recommendation at all, and higher confidence thresholds control escalation to simulation and live execution modes.
+
+Confidence is computed from four components:
+
+### Unique Source Count
+
+The function counts the number of unique document IDs across all active signals (those with `combined_weight > 0`). This count is divided by 15 and capped at `0.8`:
+
+```
+count_factor = min(unique_sources / 15.0, 0.8)
+```
+
+A trend backed by 15 or more unique source documents reaches the maximum count contribution of `0.8`. A trend backed by a single document gets only `0.067`. This component rewards breadth of evidence — a trend confirmed by many independent sources is more trustworthy than one driven by a single article, regardless of how high that article's individual weight might be.
+
+### Average Extraction Credibility
+
+The average credibility weight across all active signals provides a baseline quality measure. If most contributing documents come from high-credibility sources, this component is high. If the evidence is dominated by low-credibility sources, confidence is penalized accordingly.
+
+### Signal Agreement with Sample-Size Dampening
+
+The agreement ratio measures what fraction of directional signals (positive + negative, excluding neutral) agree on the majority direction. If 8 out of 10 directional signals are positive, the raw agreement is `0.8`. But raw agreement is misleading with small sample sizes — 1 out of 1 signals agreeing gives a perfect `1.0` agreement, which is not meaningful.
+
+To address this, the agreement is dampened by a logarithmic sample-size factor:
+
+```
+agreement_dampener = min(1.0, log₂(unique_sources + 1) / log₂(8))
+```
+
+This dampener saturates at `1.0` when `unique_sources` reaches approximately 7 (since `log₂(8) = 3.0` and `log₂(8) = 3.0`). With fewer sources, the dampener reduces the agreement contribution: 1 source gives a dampener of `0.33`, 3 sources give `0.67`, and 7 sources give the full `1.0`. The log₂ scaling means that each additional source provides diminishing marginal improvement to the dampener, which matches the intuition that the jump from 1 to 3 sources is far more meaningful than the jump from 15 to 17.
+
+### Contradiction Penalty
+
+The contradiction score computed by `services/aggregation/contradiction.py` is applied as a direct penalty:
+
+```
+contradiction_penalty = contradiction_score × 0.4
+```
+
+A contradiction score of `0.5` (perfect split) produces a penalty of `0.2`, which is substantial enough to push a moderately confident trend below the eligibility threshold.
+
+### The Combined Formula
+
+The four components are combined as:
+
+```
+confidence = 0.3 × count_factor + 0.3 × avg_credibility + 0.4 × agreement − contradiction_penalty
+```
+
+The result is clamped to `[0.0, 1.0]`. The weighting gives signal agreement the largest share (40%), reflecting the principle that consensus among diverse sources is the strongest indicator of a reliable trend. Source count and credibility each contribute 30%, providing a balanced assessment of evidence breadth and quality. The contradiction penalty can reduce confidence significantly — a highly contradicted trend with a score of 0.4 loses 0.16 points of confidence, which can easily drop it below the 0.35 eligibility gate.
+
+---
+
+## How Accumulating Signals Escalate Decisions
+
+The trend direction, strength, and confidence computed by the aggregation engine are not just descriptive — they directly determine what action the system takes. The escalation path from passive observation to active execution is governed by the eligibility thresholds defined in `services/recommendation/eligibility.py`, and the key insight is that consecutive signals pointing in the same direction naturally strengthen the trend metrics that control this escalation.
+
+### The Escalation Ladder
+
+The `EligibilityConfig` dataclass in `services/recommendation/eligibility.py` defines the thresholds that map trend metrics to actions:
+
+**Neutral (no recommendation).** A trend fails the eligibility gates entirely when confidence is below `0.35`, trend strength is below `0.10`, contradiction exceeds `0.60`, evidence count is below `2`, or the direction is neutral. The `_check_gates()` function evaluates these hard gates — if any gate fails, no recommendation is generated for that window.
+
+**Observe.** A trend that passes the gates but has a direction of mixed, or has strength below `0.25` with confidence below `0.50`, maps to an `OBSERVE` action via `_determine_action()`. This is the system's way of saying "something is happening, but the evidence is not strong enough to act on." Observe recommendations are always `informational` mode — they are logged for human review but never trigger decisions.
+
+**Monitor.** When the trend has a clear direction (positive or negative) but strength remains below `0.25` while confidence reaches `0.50` or above, the action maps to `MONITOR`. This indicates that the directional signal is real but not yet strong enough for a commitment change. Like observe, monitor recommendations are `informational` mode.
+
+**Act / Defer.** When trend strength reaches `0.25` or above with a positive direction, the action is `ACT`. With a negative direction at the same strength threshold, the action is `DEFER`. These are the only actions that can escalate beyond informational mode — `_determine_mode()` evaluates whether the recommendation qualifies for `simulation_eligible` (confidence ≥ `0.50`) or `production_eligible` (confidence ≥ `0.70`, contradiction ≤ `0.25`, evidence ≥ `5`).
+
+### How Accumulation Drives Escalation
+
+Consider an entity that starts with no recent intelligence. The first negative article arrives — a single document with negative sentiment. In the intraday window, this produces:
+
+- **Trend strength** = `|avg_sentiment|` ≈ the absolute weighted sentiment from one signal, likely close to the impact score.
+- **Confidence** = low, because `count_factor = min(1/15, 0.8) = 0.067` and the agreement dampener is only `log₂(2)/log₂(8) = 0.33`.
+- **Direction** = negative (if the weighted sentiment is ≤ -0.15).
+
+With confidence well below `0.35`, this trend fails the eligibility gate entirely. No recommendation is generated. The system is in the neutral state.
+
+A second negative article arrives hours later. Now the intraday window has two signals:
+
+- **Unique sources** = 2, so `count_factor = 0.133` and `agreement_dampener = log₂(3)/log₂(8) ≈ 0.53`.
+- **Agreement** = `1.0 × 0.53 = 0.53` (both signals agree on negative).
+- **Confidence** ≈ `0.3 × 0.133 + 0.3 × avg_cred + 0.4 × 0.53` — likely around `0.35-0.45` depending on credibility.
+
+If confidence crosses `0.35` and strength exceeds `0.10`, the trend passes the eligibility gates. But with strength below `0.25`, the action is `OBSERVE` or `MONITOR` depending on confidence.
+
+A third and fourth negative article arrive over the next day. The 1-day window now has four agreeing signals:
+
+- **Unique sources** = 4, so `count_factor = 0.267` and `agreement_dampener = log₂(5)/log₂(8) ≈ 0.77`.
+- **Agreement** = `1.0 × 0.77 = 0.77`.
+- **Confidence** ≈ `0.3 × 0.267 + 0.3 × avg_cred + 0.4 × 0.77` — likely `0.50-0.60`.
+- **Strength** = `|avg_sentiment|` — with four negative signals and no contradicting evidence, this could easily exceed `0.25`.
+
+Now the trend maps to `DEFER` with `simulation_eligible` mode (confidence ≥ `0.50`). The system has escalated from no recommendation to a simulation-eligible defer recommendation purely through the accumulation of consistent negative evidence.
+
+If the negative evidence continues — more documents, more sources, higher credibility — confidence climbs further. At confidence ≥ `0.70` with contradiction ≤ `0.25` and evidence ≥ `5`, the recommendation reaches `production_eligible` mode, the highest escalation level.
+
+The same process works in reverse for positive accumulation: consecutive favorable signals strengthen the positive trend, increase confidence through source diversity and agreement, and escalate from observe through monitor to act.
+
+### The Role of Contradiction in Preventing False Escalation
+
+Accumulation only works when signals agree. If the fifth article about an entity is positive while the previous four were negative, the contradiction score jumps — `minority_weight / total_weight` increases because the minority (positive) side now has non-zero weight. This has two effects: the contradiction penalty reduces confidence (potentially dropping it below an eligibility threshold), and if the contradiction exceeds `0.10` with `|avg_sentiment| < 0.30`, the direction flips to mixed, which maps to `OBSERVE` regardless of strength. The system effectively de-escalates when the evidence becomes contested, requiring a clearer consensus before re-escalating.
+
+---
+
+## Trend Projections
+
+After the `TrendSummary` is assembled and persisted, the aggregation engine computes a forward-looking `TrendProjection` via `compute_projection()` in `services/aggregation/projection.py`. Projections estimate where the trend is heading based on current momentum, macro signal decay, and upcoming catalysts. They are advisory — they do not directly trigger recommendations — but they provide valuable context for human reviewers and can inform future automated decision-making.
+
+### Momentum
+
+The `compute_trend_momentum()` function computes the rate of change in signed trend strength between the current and previous aggregation cycles. If the current window shows a negative trend at strength `0.40` and the previous cycle showed negative at `0.30`, the momentum is `-0.10` (strengthening negative). If no previous data is available, the function uses a heuristic: momentum is estimated as half the current signed strength, providing a reasonable baseline for new trends.
+
+Momentum enters the projection as a half-weighted adjustment to the current signed strength:
+
+```
+momentum_projected_signed = direction_sign × current_strength + momentum × 0.5
+```
+
+This means momentum influences the projection but does not dominate it — a strong current trend with weakening momentum still projects as directional, just with reduced strength.
+
+### Macro Decay
+
+The `project_macro_decay()` function estimates how active macro events will evolve over the projection horizon. Each macro event has an `estimated_duration` that maps to a decay half-life:
+
+| Duration | Half-Life |
+|----------|-----------|
+| `short_term` | 1 day |
+| `medium_term` | 7 days |
+| `long_term` | 30 days |
+
+For each event, the function computes the projected remaining impact at the end of the horizon using exponential decay: `future_factor = 2^(−future_age_days / half_life)`. The impact is further scaled by a severity weight (`critical`: 1.0, `high`: 0.75, `moderate`: 0.5, `low`: 0.25). Positive and negative macro impacts are accumulated separately, and the projected macro direction is determined by comparing the two sides — positive if the favorable side exceeds the unfavorable by 20%, negative if the reverse, mixed if both are present without a clear majority.
+
+When the macro layer is enabled and macro events exist, the projection blends the entity-specific momentum projection with the macro trajectory. The macro weight is capped at `0.4` (40% of the blended projection), ensuring that macro signals inform but do not overwhelm the entity-specific trend. The blending formula combines the signed entity projection with the signed macro projection:
+
+```
+blended = company_weight × momentum_projected + macro_weight × macro_signed
+```
+
+### Driving Factors
+
+The projection records a list of human-readable driving factors that explain what is influencing the projected direction. These include momentum descriptions ("Positive momentum (+0.150) in recent trend strength"), macro impact projections ("Macro signals project negative impact (strength 0.350) over 7d"), and upcoming catalysts drawn from the trend's `dominant_catalysts` list (limited to the top 3). If no specific factors are identified, a baseline continuation factor is recorded.
+
+### Divergence Detection
+
+After computing the projected direction, the function compares it to the current trend direction. If they differ — for example, the current trend is negative but the projection is positive due to decaying unfavorable macro events and favorable momentum — the projection is flagged with `diverges_from_current = True` and a divergence driving factor is appended. Divergence signals are particularly valuable because they indicate that the trend may be about to reverse, giving the recommendation engine and human reviewers an early warning.
+
+The projection also flags low confidence when `projected_confidence` falls below the default threshold of `0.3`. Projection confidence starts at 80% of the current trend confidence (reflecting the inherent uncertainty of forward-looking estimates), with a small boost if macro data is available and a further reduction if the macro layer is disabled entirely.
+
+---
+
+## Persistence
+
+Each aggregation cycle persists its results to four PostgreSQL tables, creating a durable record of the trend assessment and its supporting evidence.
+
+### `trend_windows` — Current State
+
+The `persist_trend_summary()` function in `services/aggregation/worker.py` upserts the `TrendSummary` into the `trend_windows` table, keyed by `(entity_type, entity_id, window)`. Each cycle overwrites the previous row for that entity and window, so `trend_windows` always reflects the most recent assessment. The row includes the trend direction, strength, confidence, contradiction score, disagreement details (as JSON), supporting and opposing evidence document IDs (as JSON arrays), dominant catalysts, material risks, environmental context, and the generation timestamp.
+
+### `trend_history` — Time-Series Snapshots
+
+Immediately after the upsert, `persist_trend_summary()` also inserts a snapshot row into the `trend_history` table. Unlike `trend_windows`, this table is append-only — every aggregation cycle adds a new row, creating a time-series of how the trend evolved over time. The history table stores the direction, strength, confidence, contradiction score, catalysts, risks, and timestamp. This time-series data powers the trend charts in the dashboard and enables the momentum computation in `services/aggregation/projection.py` by providing the previous cycle's strength and direction. If the history insert fails (for example, if the table does not yet exist in a development environment), the failure is logged at debug level and does not block the main upsert.
+
+### `trend_evidence` — Per-Document Rankings
+
+The `persist_trend_evidence()` function writes detailed evidence ranking rows to the `trend_evidence` table, linked to the `trend_windows` row by its UUID. Each row records a document ID, its role (supporting or opposing), and the individual scoring components: rank score, weight component, impact component, recency component, confidence component, and sentiment value. Non-UUID document IDs (such as synthetic pattern signal IDs like `pattern:Entity-A:performance_report:7d`) are filtered out before insertion, since the `trend_evidence` table enforces a foreign key to the `documents` table.
+
+### `trend_projections` — Forward-Looking Estimates
+
+The `persist_trend_projection()` function in `services/aggregation/projection.py` inserts the `TrendProjection` into the `trend_projections` table, linked to the `trend_windows` row. The row stores the projected direction, strength, confidence, projection horizon, driving factors (as JSON), macro contribution percentage, divergence flag, and computation timestamp. Like trend history, projections accumulate over time, allowing analysis of how well the system's forward-looking estimates matched subsequent reality.
+
+---
+
+## What Comes Next
+
+At this point, the aggregation engine has transformed weighted signals into `TrendSummary` objects across five time windows, detected contradictions, ranked evidence, computed confidence, and persisted everything to PostgreSQL. The trend metrics — direction, strength, confidence, contradiction score — encode the accumulated weight of evidence for each entity. But a `TrendSummary` is still an assessment, not an action. The next stage translates these assessments into concrete recommendations: should the system act, defer, monitor, or simply observe? And with what conviction? [Page 5 — Recommendation Generation](05-recommendation-generation.md) explains how the recommendation engine applies data quality suppression, eligibility evaluation, commitment sizing, thesis generation, and risk classification to convert trend summaries into actionable `Recommendation` objects that the decision execution engine can execute.
diff --git a/docs/sanitized-pipeline-deep-dive/05-recommendation-generation.md b/docs/sanitized-pipeline-deep-dive/05-recommendation-generation.md
new file mode 100644
index 0000000..e38b05d
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/05-recommendation-generation.md
@@ -0,0 +1,226 @@
+# Page 5 — Recommendation Generation and Signal-to-Action Translation
+
+The aggregation engine described in [Page 4](04-trend-aggregation-and-accumulating-signals.md) produces `TrendSummary` objects across five time windows for each entity identifier, encoding the direction, strength, confidence, contradiction level, and supporting evidence accumulated from all three signal layers. But a `TrendSummary` is an assessment — it describes what the evidence says, not what the system should do about it. The recommendation engine is where assessment becomes action. It takes each `TrendSummary`, subjects it to a series of deterministic evaluations, and produces a `Recommendation` object that specifies a concrete action (act, defer, monitor, or observe), an execution mode (informational, simulation-eligible, or production-eligible), a commitment sizing guideline, a human-readable thesis, and a risk classification. Every decision in this pipeline is rule-based and fully traceable — the LLM is only involved in an optional downstream step that rewrites the thesis wording.
+
+The recommendation worker in `services/recommendation/main.py` polls the `app:queue:recommendation` Redis queue for jobs, each specifying an entity identifier and time window. For each job, it delegates to `generate_recommendation()` in `services/recommendation/worker.py`, which orchestrates the full pipeline: fetch the latest trend summary, check for duplicate recommendations, fetch any available trend projection, evaluate data quality suppression, evaluate eligibility, optionally rewrite the thesis via LLM, build the `Recommendation` object, and persist everything to PostgreSQL. For a visual overview of this flow, see the [Recommendation Generation Flow diagram](diagrams/recommendation-generation-flow.md).
+
+---
+
+## Data Quality Suppression
+
+Before the eligibility engine evaluates whether a trend is strong enough to act on, the suppression layer in `services/recommendation/suppression.py` asks a more fundamental question: is the underlying data reliable enough to act on at all? A trend might show high confidence and strong directionality, but if the documents feeding it are stale, poorly extracted, or drawn from a single source type, the apparent signal quality is illusory. The suppression layer acts as a pre-filter on data quality, running before the eligibility engine and forcing any recommendation built on unreliable data to `informational` mode regardless of how strong the trend metrics look.
+
+The `evaluate_suppression()` function accepts a `TrendSummary` and a `DataQualityContext` — a set of metrics about the documents underlying the trend, populated by querying `documents` and `document_intelligence` tables for the evidence document IDs stored in the trend summary. When full document-level metrics are not available (for example, in a development environment without the full document pipeline), the function falls back to `build_quality_context_from_summary()`, which estimates quality metrics from the trend summary's own evidence counts and confidence.
+
+### The Six Data Quality Checks
+
+The suppression evaluation runs six independent checks, each comparing a data quality metric against a configurable threshold defined in `SuppressionConfig`. If any single check fails, the recommendation is suppressed:
+
+1. **Low extraction confidence** — If the average extraction confidence across the evidence documents falls below `0.40` (`min_avg_extraction_confidence`), the underlying LLM extractions are too unreliable. This catches cases where the extractor struggled with document formatting, ambiguous content, or low-quality source material, as described in [Page 2](02-ai-agent-processing-and-extraction.md).
+
+2. **Evidence staleness** — If the most recent evidence document is older than `168` hours (7 days, `max_evidence_staleness_hours`), the trend is based on outdated information. Conditions change rapidly, and a week-old evidence base may no longer reflect the current state. When documents exist but no timestamp is available, the evidence is conservatively treated as stale.
+
+3. **Low source diversity** — If fewer than `1` distinct source type (`min_source_types`) contributed to the evidence, the signal may be driven by a single unreliable source class. In practice, this check fires when the quality context has documents but all come from the same source type (for example, all news articles with no filings or supplementary data to corroborate).
+
+4. **High extraction failure rate** — If more than `50%` (`max_extraction_failure_rate`) of the documents that should have contributed to the trend failed extraction entirely, the data pipeline is unreliable for this entity. A high failure rate means the trend summary is built from a biased subset of the available evidence — the failed documents might have told a different story.
+
+5. **Insufficient valid documents** — If fewer than `2` valid (non-failed) documents (`min_valid_documents`) contributed to the trend, there simply is not enough data to act on. A single document, no matter how high-quality, does not provide the corroboration needed for automated execution decisions.
+
+6. **Low data quality score** — The `_compute_data_quality_score()` function computes an overall quality score from three weighted components: extraction confidence (40% weight, normalized against a 0.8 baseline), evidence freshness (30% weight, linear decay over the staleness window), and document coverage (30% weight, combining the valid/total ratio with a count factor that saturates at 10 documents). If this composite score falls below `0.30` (`min_data_quality_score`) and the low-confidence check has not already fired, a general suppression reason is added.
+
+When any check triggers, the `SuppressionResult` records the specific reasons (as `SuppressionReason` enum values) and the computed data quality score. The worker in `services/recommendation/worker.py` uses this result to force the recommendation's mode to `informational` and append a suppression note to the thesis text, ensuring the suppression decision is visible in the audit trail.
+
+### Safety Suppressions: Macro-Only and Pattern-Only Signals
+
+Beyond the six data quality checks, two additional safety suppressions protect against acting on signals that lack entity-specific corroboration:
+
+**Macro-only suppression** (`evaluate_macro_only_suppression()`) fires when macro signals are the sole basis for a trend direction — no entity-specific signals contributed at all. As described in [Page 3](03-signal-scoring-and-weighted-signals.md), macro signals enter the aggregation engine at a reduced weight of `0.3` relative to entity-specific signals. But even at reduced weight, macro signals alone can shift a trend direction if no entity-specific evidence exists. When this happens, the recommendation is forced to `informational` mode with a caveat noting that the signal is macro-only and should not be used for automated execution.
+
+**Pattern-only suppression** (`evaluate_pattern_only_suppression()`) applies the same logic to competitive/pattern signals. When pattern-based signals from `services/aggregation/pattern_matcher.py` and `services/aggregation/signal_propagation.py` are the sole contributors — no entity-specific or macro signals — the recommendation is suppressed. Historical patterns are valuable context, but acting on them without any current evidence is too speculative for automated execution.
+
+Both safety suppressions are evaluated in the worker after the main suppression check, and both force the mode to `informational` when triggered.
+
+---
+
+## Eligibility Evaluation
+
+Recommendations that survive the suppression layer enter the eligibility evaluation in `services/recommendation/eligibility.py`. This is the core decision logic — a set of deterministic rules that map trend metrics to actions, execution modes, and commitment sizing. The `evaluate_eligibility()` function is the single entry point, accepting a `TrendSummary` and an `EligibilityConfig` of tunable thresholds.
+
+### Gate Checks
+
+The `_check_gates()` function applies five hard gates. If any gate fails, the trend is ineligible for a recommendation (though the action and mode are still computed for the audit trace):
+
+| Gate | Threshold | Rejection Reason |
+|------|-----------|-----------------|
+| Confidence | ≥ `0.35` | `low_confidence` |
+| Trend strength | ≥ `0.10` | `low_trend_strength` |
+| Contradiction score | ≤ `0.60` | `high_contradiction` |
+| Evidence count | ≥ `2` (supporting + opposing) | `insufficient_evidence` |
+| Direction | ≠ `neutral` | `neutral_direction` |
+
+These gates are intentionally conservative. A confidence threshold of `0.35` means the system needs meaningful evidence breadth and agreement before generating any recommendation at all (see the confidence computation in [Page 4](04-trend-aggregation-and-accumulating-signals.md)). The contradiction ceiling of `0.60` allows moderately contested trends through — only when the evidence is deeply split does the gate reject. The evidence minimum of `2` ensures that no recommendation is ever based on a single document.
+
+When a trend fails any gate, the resulting `EligibilityResult` has `eligible = False` and the mode is forced to `informational`, regardless of what the mode escalation logic would otherwise compute.
+
+### Action Mapping
+
+The `_determine_action()` function maps the trend's direction and strength to one of four action types. The logic evaluates in a specific order:
+
+**Mixed or neutral direction → OBSERVE.** If the trend direction is `mixed` (high contradiction with weak directional signal) or `neutral`, the action is always `OBSERVE`. There is no directional conviction to act on.
+
+**Strong directional signal → ACT or DEFER.** If the trend strength reaches `0.25` or above (`action_strength_threshold`), the action follows the direction: `ACT` for positive, `DEFER` for negative. This threshold ensures that only trends with meaningful magnitude trigger commitment-changing actions.
+
+**Weak directional signal with decent confidence → MONITOR.** If the trend has a clear direction (positive or negative) but strength remains below `0.25`, the action depends on confidence. If confidence reaches `0.50` or above (`hold_confidence_threshold`), the action is `MONITOR` — the system recognizes the directional lean but does not have enough conviction to recommend a commitment change. Below `0.50` confidence, the action falls to `OBSERVE`.
+
+This mapping creates the escalation ladder described in [Page 4](04-trend-aggregation-and-accumulating-signals.md): as consecutive signals accumulate and strengthen the trend metrics, the action naturally progresses from OBSERVE → MONITOR → ACT/DEFER.
+
+### Mode Escalation
+
+The `_determine_mode()` function determines the highest execution mode allowed for the recommendation. Mode controls whether the recommendation is purely informational, eligible for simulation mode, or eligible for live execution mode:
+
+**OBSERVE and MONITOR → always informational.** These actions do not trigger executions, so they are always `informational` mode. They are logged for human review and dashboard display but never enter the decision execution engine.
+
+**ACT and DEFER → escalation based on signal quality.** For actionable recommendations, mode escalates through three tiers:
+
+- **`informational`** — The default when confidence is below `0.50`. The recommendation is recorded but not eligible for any execution.
+- **`simulation_eligible`** — When confidence reaches `0.50` or above (`paper_confidence_threshold`). The recommendation can be picked up by the simulation engine described in [Page 6](06-decision-execution.md).
+- **`production_eligible`** — The strictest tier, requiring confidence ≥ `0.70` (`live_confidence_threshold`), contradiction ≤ `0.25` (`live_max_contradiction`), and evidence count ≥ `5` (`live_min_evidence`). This triple gate ensures that only high-conviction, well-corroborated, low-contradiction recommendations can trigger live executions.
+
+The evidence count for mode escalation is computed as the sum of supporting and opposing evidence documents, matching the same count used in the gate checks.
+
+---
+
+## Commitment Sizing
+
+The `_compute_position_sizing()` function in `services/recommendation/eligibility.py` translates signal quality into an allocation pool guideline. Commitment sizing is not a fixed value — it scales dynamically with the confidence and strength of the underlying trend, penalized by contradiction and thin evidence.
+
+### Base and Scaling
+
+The computation starts with a base allocation of `1%` (`base_allocation_pct = 0.01`) and scales upward based on two factors:
+
+- **Confidence factor** — `0.8 × confidence` (`confidence_sizing_weight`), reflecting how much the system trusts the trend assessment.
+- **Strength factor** — `0.5 + 0.5 × trend_strength`, ranging from `0.5` (weakest trend) to `1.0` (strongest trend).
+
+The raw allocation percentage is computed as:
+
+```
+raw_allocation = base + confidence_factor × strength_factor × (max - base)
+```
+
+where `max` is `10%` (`max_allocation_pct = 0.10`). At maximum confidence (1.0) and maximum strength (1.0), the raw allocation reaches the full 10%. At typical values (confidence 0.6, strength 0.3), the raw allocation is considerably lower.
+
+### Contradiction Penalty
+
+The contradiction score applies a multiplicative penalty:
+
+```
+allocation_pct = raw_allocation × (1.0 − 0.5 × contradiction_score)
+```
+
+A contradiction score of `0.40` reduces the allocation by 20%. A score of `0.0` (no contradiction) applies no penalty. This ensures that contested trends receive smaller commitment sizes even when they pass the eligibility gates.
+
+### Evidence Count Penalty
+
+Thin evidence further reduces the allocation:
+
+- Fewer than `3` evidence documents → multiply by `0.5` (halved).
+- Fewer than `5` evidence documents → multiply by `0.75`.
+- `5` or more documents → no penalty.
+
+This penalty stacks with the contradiction penalty, so a trend with high contradiction and thin evidence receives a substantially reduced commitment size.
+
+### Max Loss Scaling
+
+The same scaling logic applies to the maximum loss percentage, which starts at a base of `0.3%` (`base_max_loss_pct = 0.003`) and scales up to `2%` (`max_max_loss_pct = 0.02`). Higher-conviction commitments are allowed larger loss tolerances, while low-conviction or contested commitments are constrained to tighter risk thresholds.
+
+The final `PositionSizing` object (defined in `services/shared/schemas.py`) contains `allocation_pct` and `max_loss_pct`, both clamped to their respective bounds. This object is embedded in the `Recommendation` and later consumed by the decision execution engine's own commitment sizer (described in [Page 6](06-decision-execution.md)), which applies additional resource pool-level constraints.
+
+---
+
+## Thesis Generation
+
+Every recommendation includes a human-readable thesis that explains the reasoning behind the action. Thesis generation happens in two layers: a deterministic assembly that is always present, and an optional LLM rewrite that polishes the wording for execution-eligible recommendations.
+
+### Deterministic Thesis Assembly
+
+The `build_thesis()` function in `services/recommendation/worker.py` constructs a thesis string entirely from the trend data and eligibility result, with no model involvement. The thesis is assembled from several components in order:
+
+1. **Opening** — States the entity identifier, trend direction, window, strength, and confidence. For example: "Entity-A shows a negative trend over the 7d window with strength 0.35 and confidence 0.62."
+
+2. **Catalysts** — Lists the top three dominant catalysts from the `TrendSummary`, drawn from the evidence ranking described in [Page 4](04-trend-aggregation-and-accumulating-signals.md).
+
+3. **Contradiction note** — If the contradiction score exceeds `0.15`, a note flags the signal disagreement and its magnitude.
+
+4. **Trend projection** — When a `TrendProjection` is available and not flagged as low-confidence, the thesis incorporates the projected direction, strength, and top driving factors. If the projection diverges from the current trend, a divergence note is appended.
+
+5. **Risks** — Lists the top two material risks from the `TrendSummary`.
+
+6. **Evidence count** — States the number of supporting and opposing evidence documents.
+
+7. **Prescriptive action** — States the recommended action and mode (e.g., "Recommendation: DEFER (simulation eligible).").
+
+The deterministic thesis is always generated and serves as the audit reference. Even when the LLM rewrites the thesis, the deterministic version is preserved in the model metadata for traceability.
+
+### Optional LLM Rewrite via the Thesis-Rewriter Agent
+
+For recommendations that are both eligible and not suppressed, the worker optionally invokes the thesis-rewriter agent to polish the deterministic thesis into professional-quality prose. The LLM rewrite is implemented in `services/recommendation/thesis_llm.py` and uses the `thesis-rewriter` agent slug, resolved at runtime through the `AgentConfigResolver` in `services/shared/agent_config.py`.
+
+The `AgentConfigResolver` queries the `ai_agents` and `agent_variants` database tables to resolve the active configuration for the `thesis-rewriter` slug, preferring an active variant's model, timeout, and retry settings when one exists. The resolver uses a 60-second TTL in-memory cache to avoid hitting the database on every recommendation. This is the same resolution mechanism used by the document extractor and event classifier agents described in [Page 2](02-ai-agent-processing-and-extraction.md).
+
+The `rewrite_thesis_with_llm()` function builds a prompt from the deterministic thesis and trend context (entity identifier, window, direction, strength, confidence, contradiction score, catalysts, risks), sends it to the local Ollama instance via HTTP, and returns the rewritten text. The system prompt enforces strict rules: no fabricated information, no numbers or facts not present in the input, under 150 words, neutral professional tone, and only the rewritten thesis text in the response.
+
+The LLM layer is purely additive — if the call fails for any reason (network error, timeout, empty response, token budget exceeded), the original deterministic thesis is returned unchanged. The worker in `services/recommendation/main.py` resolves the thesis-rewriter configuration at startup and refreshes it every 50 jobs to pick up configuration changes without requiring a restart. When no database configuration exists for the `thesis-rewriter` slug, thesis rewriting is silently disabled.
+
+Performance logging for the thesis-rewriter is written to the `agent_performance_log` table, recording success/failure, duration, estimated token counts, and the variant ID. Token budget enforcement checks hourly usage against the variant's configured budget before making the LLM call, preventing runaway costs from high-volume recommendation cycles.
+
+### Risk Classification Prefix
+
+Before the thesis is stored, the `classify_risk()` function in `services/recommendation/worker.py` assigns a risk classification label that is prepended to the thesis text as a `[risk:<level>]` prefix. The classification is computed from a composite score:
+
+| Factor | Contribution |
+|--------|-------------|
+| Contradiction score | `contradiction × 2.0` |
+| Low confidence | `(1.0 − confidence) × 1.5` |
+| Low evidence count | `+1.0` if < 3 docs, `+0.5` if < 5 docs |
+| Rejection reasons | `+0.5` per rejection reason |
+
+The composite score maps to four levels:
+
+| Score Range | Classification |
+|-------------|---------------|
+| ≥ 3.0 | `very_high` |
+| ≥ 2.0 | `high` |
+| ≥ 1.0 | `moderate` |
+| < 1.0 | `low` |
+
+A recommendation with high contradiction (0.4 → contributes 0.8), moderate confidence (0.55 → contributes 0.675), and 4 evidence documents (contributes 0.5) would score 1.975, classifying as `moderate`. The same recommendation with only 2 evidence documents would score 2.475, pushing it to `high`. This classification gives downstream consumers — both the decision execution engine and human reviewers — a quick risk signal without needing to re-evaluate the underlying metrics.
+
+---
+
+## Persistence
+
+The recommendation pipeline persists its output to three PostgreSQL tables, creating a complete audit trail from trend assessment through decision logic to the final recommendation.
+
+### `recommendations` — The Core Record
+
+The `persist_recommendation()` function in `services/recommendation/worker.py` inserts the `Recommendation` into the `recommendations` table. Each row captures the entity identifier, action, mode, confidence, time horizon, thesis (including the risk classification prefix and any suppression notes), invalidation conditions (as JSONB), commitment sizing (allocation percentage and max loss percentage), model metadata (provider, model name, prompt version, schema version), risk classification, and generation timestamp. The insert returns the recommendation's UUID, which serves as the foreign key for the evidence and risk evaluation tables.
+
+### `recommendation_evidence` — Evidence Citations
+
+For each evidence document referenced in the recommendation, a row is inserted into the `recommendation_evidence` table linking the recommendation UUID to the document UUID, with an evidence type (`supporting` or `opposing`) and a position-based weight that decays with rank: `weight = 1.0 / (1.0 + index × 0.1)`. The first supporting document gets weight `1.0`, the second gets `0.91`, the third `0.83`, and so on. Non-UUID document IDs (such as synthetic pattern signal IDs like `pattern:Entity-A:performance_report:7d` from the competitive signal layer) are filtered out before insertion, since the table enforces a foreign key to the `documents` table.
+
+### `risk_evaluations` — Decision Audit Trail
+
+The `risk_evaluations` table records the full eligibility decision for each recommendation: whether the trend was eligible, the allowed mode, the list of rejection reasons (as JSONB), and a `risk_checks` JSONB object containing the time horizon, commitment sizing details, invalidation conditions, and risk classification. This table enables post-hoc analysis of why the system made a particular decision — auditors can trace from the recommendation back through the eligibility evaluation to the underlying trend metrics.
+
+---
+
+## Deduplication
+
+Before running the full evaluation pipeline, the worker checks whether the latest recommendation for the same entity identifier and time horizon is effectively identical to what would be generated. The `_is_duplicate_recommendation()` function in `services/recommendation/worker.py` compares the previous recommendation's action, mode, and confidence (within a `0.01` tolerance) against the current eligibility result. If all three match, the recommendation is skipped — the underlying trend data has not changed meaningfully since the last cycle. This prevents the system from flooding the `recommendations` table with identical entries on every aggregation cycle, while still generating a new recommendation whenever the trend metrics shift enough to change the action, mode, or confidence.
+
+---
+
+## What Comes Next
+
+At this point, the recommendation engine has translated trend assessments into concrete `Recommendation` objects — each with an action, execution mode, commitment sizing guideline, thesis, and risk classification — and persisted them alongside their evidence citations and eligibility audit trails. Recommendations marked as `simulation_eligible` or `production_eligible` are now available for the decision execution engine to consume. [Page 6 — Decision Execution](06-decision-execution.md) explains how the decision execution engine polls these recommendations, applies its own pre-execution check sequence (circuit breakers, execution windows, confidence gates, deduplication, declining commitments, and max open commitments), computes final commitment sizes with resource pool-level constraints, and submits execution requests through the execution adapter to the external execution API.
\ No newline at end of file
diff --git a/docs/sanitized-pipeline-deep-dive/06-decision-execution.md b/docs/sanitized-pipeline-deep-dive/06-decision-execution.md
new file mode 100644
index 0000000..9a45219
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/06-decision-execution.md
@@ -0,0 +1,199 @@
+# Page 6 — Decision Execution
+
+The recommendation engine described in [Page 5](05-recommendation-generation.md) produces `Recommendation` objects with an action, execution mode, commitment sizing guideline, thesis, and risk classification. Recommendations marked as `simulation_eligible` or `production_eligible` are persisted to the `recommendations` table and are now available for the final stage of the pipeline: autonomous decision execution. The decision execution engine in `services/trading/engine.py` is where intelligence becomes action. It polls eligible recommendations, subjects each one to a strict sequence of pre-execution safety checks, computes a pool-aware commitment size, and — if every gate passes — submits an execution request through the execution adapter to the external execution API. Every evaluation, whether it results in a decision or a skip, is recorded as a `DecisionRecord` in the `execution_decisions` table, creating a complete audit trail from the original document signal through to the execution response.
+
+For a visual overview of the decision flow, see the [Decision Engine Loop diagram](diagrams/decision-engine-loop.md).
+
+---
+
+## The Decision Execution Engine Loop
+
+The `DecisionEngine` class in `services/trading/engine.py` is the orchestrator. When `start()` is called, it loads the current resource pool state from PostgreSQL — active commitments, reserve pool balance, sector exposure, pool exposure — and then spawns five concurrent `asyncio` tasks that run for the lifetime of the engine:
+
+1. **`_decision_loop()`** — The core polling loop. Every 60 seconds (configurable via `polling_interval_seconds`), it queries the `recommendations` table for rows where `action IN ('act', 'defer')`, `mode IN ('simulation_eligible', 'production_eligible')`, and `generated_at` is within the last two hours. Recommendations are ordered by confidence descending and capped at 50 per cycle. For each recommendation, the engine fetches the current data point (first from `market_snapshots`, falling back to the data source API), then runs the full pre-execution evaluation pipeline described below.
+
+2. **`_risk_threshold_monitor()`** — Periodically checks current values against the risk threshold and gain target levels maintained by the `RiskThresholdManager` in `services/trading/stop_loss_manager.py`. When a value crosses a risk threshold or gain target, the monitor submits a defer execution request to the execution queue. The `RiskThresholdManager` computes initial levels from ATR and risk tier parameters, re-evaluates them when volatility shifts materially (ATR change > 10%), activates trailing thresholds when the value moves more than 50% toward the gain target, and tightens thresholds proactively when pool exposure exceeds 80% of the maximum.
+
+3. **`_performance_loop()`** — Computes pool-wide performance metrics (total value, unrealized and realized gain/loss, success rate, risk-adjusted return ratio, peak-to-trough decline, pool exposure), persists daily snapshots to `pool_snapshots`, checks for daily-loss circuit breaker triggers, evaluates gain-taking opportunities, and synchronizes commitments with the database to detect closed commitments and trigger reserve pool siphoning.
+
+4. **`_risk_tier_scheduler()`** — Runs once daily at 16:00 ET (session close). It loads the latest `PerformanceMetrics` from `pool_snapshots`, computes the reserve pool as a fraction of total resource pool value, and delegates to the `RiskTierController` in `services/trading/risk_tier_controller.py` to determine whether the active risk tier should change. Tier changes are persisted to `risk_tier_history` and take effect immediately for subsequent decision cycles.
+
+5. **`_rebalance_scheduler()`** — Runs weekly on Monday at 09:45 ET (shortly after session open). It loads current commitments, evaluates them against the active risk tier's constraints using the `PoolRebalancer`, and pushes any rebalance defer execution requests to `app:queue:execution_orders`. The rebalancer respects the circuit breaker — if any breaker is active, the rebalance cycle is skipped entirely.
+
+All five tasks run concurrently within a single `asyncio` event loop. Graceful shutdown via `stop()` cancels all tasks and awaits their completion. If any task encounters an unexpected exception, it logs the error and retries after a brief sleep rather than crashing the engine.
+
+---
+
+## Pre-Execution Check Sequence
+
+When the decision loop picks up an act recommendation, it calls `evaluate_recommendation()` — a synchronous method that runs the full pre-execution check sequence. The checks are applied in a strict order, and the first failure short-circuits the evaluation with a `skip` decision. This fail-fast design ensures that expensive downstream computations (like commitment sizing and correlation analysis) are never reached when a simple gate would have rejected the decision.
+
+The six checks, in order:
+
+**a. Circuit breaker check.** The engine calls `self.circuit_breaker.is_active()` on the current `CircuitBreakerState`. If any circuit breaker is active and its cooldown has not expired, the recommendation is skipped with reason `circuit_breaker_active`. The circuit breaker mechanism is described in detail below.
+
+**b. Execution window check.** The `is_within_execution_window()` function verifies that the current time falls within the active session hours. Outside the execution window, no execution requests are submitted — the recommendation is skipped with reason `outside_execution_window`.
+
+**c. Confidence gate.** The recommendation's confidence score is compared against the active risk tier's `min_confidence` threshold. A conservative tier requires confidence ≥ 0.75, moderate requires ≥ 0.55, and aggressive requires ≥ 0.40. If the recommendation's confidence falls below the tier minimum, it is skipped with reason `insufficient_confidence`. This gate ensures that the risk tier's conservatism is enforced before any resource allocation is considered.
+
+**d. Deduplication check.** The engine maintains an in-memory set of processed recommendation IDs (`processed_recommendation_ids`) and also checks Redis via `app:dedupe:execution:*` keys (with a 24-hour TTL). If the recommendation has already been evaluated in this engine session or by a previous instance, it is skipped with reason `duplicate_recommendation`. This prevents the same recommendation from generating multiple execution requests across polling cycles.
+
+**e. Declining commitments check.** The `check_declining_commitments()` method examines all active commitments. If more than 50% of commitments have unrealized losses exceeding 2% of their entry value, the engine halts new entries with reason `multiple_declining_commitments`. This is a pool-level safety valve — when the majority of existing commitments are underwater, adding new exposure compounds the risk.
+
+**f. Max active commitments check.** The engine enforces a configurable maximum number of concurrent commitments (default 10). If the resource pool is already at capacity, the recommendation is skipped with reason `max_commitments_reached`.
+
+For defer recommendations, the engine follows a separate, simpler path: it verifies the execution window, looks up the existing commitment for the entity, and submits a full-quantity defer execution request without running the commitment sizer. Defer decisions still generate an audit record in `execution_decisions` and set the Redis deduplication key.
+
+If all six checks pass for an act recommendation, the engine proceeds to commitment sizing.
+
+---
+
+## Commitment Sizing
+
+The `CommitmentSizer` in `services/trading/position_sizer.py` translates a recommendation's signal quality into a concrete dollar amount and unit count, applying a sequential pipeline of adjustments that account for confidence, pool composition, sector concentration, correlation, and upcoming performance report events. The sizer operates on the *active pool* — the portion of the resource pool available for execution after subtracting the reserve pool balance.
+
+### Base Sizing
+
+The computation begins with a base allocation percentage derived from the risk tier:
+
+```
+base_allocation_pct = risk_tier.max_position_pct × 0.5
+raw_pct = base_allocation_pct × (confidence / min_confidence)
+```
+
+The base starts at half the tier's maximum commitment percentage, then scales linearly with how far the recommendation's confidence exceeds the tier minimum. A moderate-tier recommendation with confidence 0.70 against a minimum of 0.55 would produce a raw percentage of `0.05 × (0.70 / 0.55) ≈ 0.0636`, or about 6.4% of the active pool. The raw percentage is clamped to `max_position_pct` (5% for conservative, 10% for moderate, 15% for aggressive) and then converted to a dollar amount against the active pool. An absolute commitment cap (default $50) provides a hard ceiling regardless of pool size — a safety measure for the simulation mode environment.
+
+### Correlation-Aware Diversification
+
+The sizer computes a weighted average correlation between the candidate entity and all existing commitments, using the pairwise correlation matrix that the engine refreshes from 30 days of daily close values in `market_snapshots`. Each existing commitment's correlation is weighted by its value, so larger commitments have more influence on the diversification check.
+
+If the weighted average correlation exceeds 0.8, the commitment is rejected outright — the resource pool already has too much exposure to correlated assets. Between 0.5 and 0.8, the dollar amount is reduced proportionally: a correlation of 0.65 produces a scale factor of `1.0 − (0.65 − 0.5) / (0.8 − 0.5) = 0.5`, halving the commitment size. Below 0.5, no reduction is applied.
+
+### Sector Exposure Reduction
+
+The sizer checks whether adding the new commitment would push the sector's total exposure beyond the risk tier's `max_sector_pct` (20% for conservative, 30% for moderate, 40% for aggressive). If the sector is already at its limit, the commitment is rejected. If the new commitment would exceed the limit, the dollar amount is reduced to exactly fill the remaining sector capacity.
+
+### Diversification Bonus
+
+When the resource pool holds fewer than three distinct sectors and the candidate entity belongs to a new sector, the sizer applies a 1.2× bonus to the dollar amount. This incentivizes early diversification — the first few commitments are encouraged to spread across sectors rather than concentrating in a single one. The bonus is re-clamped to `max_position_pct` after application to prevent oversized commitments.
+
+### Performance Report Proximity Adjustment
+
+The sizer checks the performance report calendar for the candidate entity. If a performance report is within one active session, the commitment is rejected entirely — the binary risk of a disclosure surprise is too high for automated entry. If a performance report is within three active sessions, the dollar amount is reduced by 50%. Beyond three sessions, no adjustment is applied.
+
+### Pool Exposure Check and Unit Rounding
+
+After all adjustments, the sizer estimates the new commitment's contribution to pool exposure (the aggregate risk from risk threshold distances across all commitments). If adding the commitment would push total exposure beyond `max_portfolio_heat × active_pool` (10% for conservative, 20% for moderate, 30% for aggressive), the commitment is rejected.
+
+Finally, the dollar amount is converted to whole units via `floor(dollar_amount / current_value)`. If rounding produces zero units (the commitment is too small for even one unit at the current value), the commitment is rejected. The final dollar amount is recalculated from the whole-unit quantity to reflect the actual capital deployed.
+
+The `CommitmentSizeResult` returned to the engine includes the dollar amount, unit quantity, allocation percentage, a list of human-readable adjustment notes, and a rejected flag with reason if any step failed. These adjustment notes are embedded in the decision record's `decision_trace` for full auditability.
+
+---
+
+## Circuit Breaker
+
+The `CircuitBreaker` in `services/trading/circuit_breaker.py` is a pure computation module that evaluates three independent trigger conditions. It carries no state of its own — the engine manages the `CircuitBreakerState` dataclass and persists trigger events to the `circuit_breaker_events` table and Redis keys under `app:execution:circuit_breaker:*`.
+
+### Three Trigger Types
+
+**Daily loss trigger.** When the resource pool's daily gain/loss exceeds 5% of total resource pool value (`daily_loss_pct = 0.05`), the circuit breaker activates. The `check_daily_loss()` method compares the absolute loss ratio against the threshold. The cooldown duration is set to `volatility_pause_hours` (default 2 hours). The performance loop in the engine calls `_check_circuit_breaker_daily_loss()` periodically to evaluate this condition against the latest pool metrics. In extreme cases where the peak-to-trough decline exceeds an emergency threshold, the reserve pool's emergency liquidation mechanism may also be triggered.
+
+**Single commitment loss trigger.** When any individual commitment loses more than 15% of its entry value (`single_position_loss_pct = 0.15`), the circuit breaker activates with an entity-specific cooldown. The `check_single_position()` method evaluates the loss percentage. The cooldown for the affected entity is set to `ticker_cooldown_hours` (default 48 hours), during which the engine will not re-enter that entity. The `is_ticker_cooled_down()` method checks whether a specific entity is still within its cooldown window by consulting the `ticker_cooldowns` dictionary in the `CircuitBreakerState`.
+
+**Volatility trigger (risk threshold clustering).** When three or more risk thresholds fire within a 30-minute rolling window (`stop_loss_hits_threshold = 3`, `stop_loss_window_minutes = 30`), the circuit breaker activates. The `check_volatility()` method uses a sliding window algorithm: it sorts the risk threshold timestamps and checks every contiguous subsequence of length `stop_loss_hits_threshold` to see if it fits within the window. This detects rapid-fire risk threshold cascades that indicate extreme volatility. The cooldown is `volatility_pause_hours` (default 2 hours).
+
+### Cooldown Computation
+
+The `compute_cooldown_expiry()` method calculates when a triggered breaker expires. For `daily_loss` and `volatility` triggers, the expiry is `triggered_at + volatility_pause_hours`. For `single_position` triggers, the expiry is `triggered_at + ticker_cooldown_hours`, giving the affected entity a longer cooling-off period. The `is_active()` method returns `True` when the breaker is flagged active and the current time has not yet passed the cooldown expiry.
+
+### Redis State Tracking
+
+The engine persists circuit breaker state to Redis under the `app:execution:circuit_breaker:*` key pattern (constructed by `execution_cb_key()` in `services/shared/redis_keys.py`). Each trigger type gets its own key — for example, `app:execution:circuit_breaker:daily_loss` — storing the activation timestamp and cooldown expiry. This allows the state to survive engine restarts and enables external monitoring tools to query breaker status without accessing the engine's memory.
+
+---
+
+## Reserve Pool
+
+The `ReservePoolController` in `services/trading/reserve_pool.py` manages an untouchable cash reserve that grows from realized execution gains. The reserve serves two purposes: it provides a buffer against peak-to-trough declines, and its size relative to the resource pool influences risk tier upgrade decisions.
+
+### Profit Siphoning
+
+When the engine detects a closed commitment with positive unrealized gain/loss (via `_sync_commitments_and_siphon()` in the performance loop), it calls `siphon_profit()` on the controller. The method transfers a configurable fraction of the realized gain into the reserve — by default 20% (`siphon_pct = 0.20`). Only positive gains are siphoned; losses do not reduce the reserve balance. Each siphon event is recorded in the `reserve_pool_ledger` table with the transfer amount, resulting balance, trigger type (`profit_siphon`), the entity as reference, and a timestamp.
+
+### High-Water Mark Rebalancing
+
+The `is_high_water()` method returns `True` when the reserve balance exceeds 30% of total resource pool value (`high_water_pct = 0.30`). This signal is consumed by the risk tier scheduler — when the reserve is healthy and other performance criteria are met, the controller may recommend upgrading to a more aggressive tier. The high-water mark acts as a confidence indicator: a large reserve means the system has been consistently successful and can afford to take on more risk.
+
+### Emergency Liquidation
+
+The `should_emergency_liquidate()` method checks whether the current peak-to-trough decline exceeds an emergency threshold. When triggered, `emergency_liquidate()` returns the full reserve balance for release back into the active pool. The caller (the engine) is responsible for zeroing the persisted balance and recording the ledger entry. Emergency liquidation is a last resort — it sacrifices the safety buffer to prevent the resource pool from hitting a catastrophic loss level.
+
+### Active Pool Computation
+
+The `compute_active_pool()` method calculates the capital available for execution: `active_pool = total_pool_value − reserve_balance`. All commitment sizing computations use the active pool rather than the total resource pool value, ensuring that the reserve is never inadvertently deployed into new commitments.
+
+---
+
+## Risk Tier Auto-Adjustment
+
+The `RiskTierController` in `services/trading/risk_tier_controller.py` evaluates resource pool performance and determines whether the active risk tier should shift. The system supports three tiers — conservative, moderate, and aggressive — each defined by a `RiskTierConfig` dataclass in `services/trading/models.py` with distinct parameter values:
+
+| Parameter | Conservative | Moderate | Aggressive |
+|-----------|-------------|----------|------------|
+| `min_confidence` | 0.75 | 0.55 | 0.40 |
+| `max_position_pct` | 5% | 10% | 15% |
+| `stop_loss_atr_multiplier` | 1.5× | 2.0× | 2.5× |
+| `reward_risk_ratio` | 2.0 | 1.5 | 1.2 |
+| `max_sector_pct` | 20% | 30% | 40% |
+| `max_portfolio_heat` | 10% | 20% | 30% |
+
+The tier controller's `evaluate()` method checks two conditions:
+
+**Downgrade (any one triggers).** If the trailing 30-day success rate drops below 40% or the current peak-to-trough decline exceeds 15%, the tier steps down by one level (e.g., aggressive → moderate). If the system is already at conservative, no further downgrade is possible.
+
+**Upgrade (all must be true).** If the success rate exceeds 55%, the reserve pool exceeds 20% of total resource pool value, and the current peak-to-trough decline is below 5%, the tier steps up by one level. The triple requirement ensures that upgrades only happen when the system is performing well, has built a safety cushion, and is not in a decline.
+
+The risk tier scheduler in the engine evaluates these conditions daily at session close. When a tier change occurs, it is persisted to the `risk_tier_history` table with the previous tier, new tier, trigger source (`auto_adjustment`), and the metrics that drove the decision (success rate, peak-to-trough decline, reserve percentage, risk-adjusted return ratio). The new tier takes effect immediately — the engine updates its `_active_risk_tier` reference, and all subsequent decision cycles use the new tier's parameters for confidence gates, commitment sizing, risk threshold computation, and sector exposure limits.
+
+---
+
+## Execution Request Submission Flow
+
+When `evaluate_recommendation()` returns an `act` decision, the engine constructs an execution request job and pushes it through a multi-stage submission pipeline that spans two services.
+
+### Decision Persistence
+
+Every evaluation — whether it results in `act` or `skip` — produces a decision record that is persisted to the `execution_decisions` table via `_persist_decision()`. The record captures the recommendation ID, decision outcome, skip reason (if applicable), entity identifier, computed commitment size and unit quantity, the risk tier at the time of decision, pool exposure, active pool and reserve pool balances, circuit breaker status, correlation and sector exposure check results, performance report proximity flag, and a `decision_trace` JSONB field containing the full reasoning chain. This creates a complete audit record of every recommendation the engine evaluated and why it acted or declined.
+
+### Execution Request Enqueue
+
+For `act` decisions, the engine builds an execution request job dictionary containing the decision ID, entity identifier, action (act or defer), quantity, and request type (immediate). This job is pushed via `rpush` to the `app:queue:execution_orders` Redis queue (constructed by `queue_key(QUEUE_BROKER)` from `services/shared/redis_keys.py`). The engine immediately deducts the estimated execution cost from the in-memory active pool to prevent over-allocation across concurrent recommendation evaluations within the same polling cycle.
+
+### Execution Service Processing
+
+The execution service in `services/adapters/broker_service.py` runs as a standalone worker that polls `app:queue:execution_orders` via `blpop`. For each execution request job, `process_order_job()` executes a multi-step pipeline:
+
+1. **Idempotency check.** A deterministic idempotency key is generated from the job's entity identifier, action, quantity, and decision ID. The service checks Redis first (fast path) and then the `orders` table (durable fallback) to prevent duplicate submissions. If a matching key exists, the job is silently dropped.
+
+2. **Risk evaluation.** The service loads the current `PoolRiskConfig` from the database and the account's risk state (active commitments, daily gain/loss, sector exposure) from both the database and the external execution API. The `evaluate_order()` function runs the proposed execution request through a set of risk checks — commitment limits, sector concentration, daily loss thresholds — and produces an evaluation result. The evaluation is persisted to the `risk_evaluations` table regardless of outcome.
+
+3. **External API submission.** If the risk evaluation passes, the service calls `submit_order()` on the `ExecutionAdapter` in `services/adapters/broker_adapter.py`. The adapter constructs the external execution API payload (entity identifier, quantity, side, request type, time in force) and submits it to `execution-api.example.com/v2/orders` with an idempotency key header. The adapter follows a fail-closed policy: any network error or ambiguous response returns a rejected `ExecutionResponse` rather than risking duplicate execution requests.
+
+4. **Persistence and audit trail.** The `persist_order()` function writes the execution request to the `orders` table with the full request and response details, risk evaluation results, and the recommendation ID for traceability. When the execution request is filled, the fill details (value, quantity) are recorded. Execution request events are published to the analytical lakehouse via MinIO for downstream analysis. The Redis idempotency marker is set after successful persistence to prevent reprocessing.
+
+The result is a complete chain of custody: from the original document that produced a signal (Pages [1](01-data-ingestion-and-preparation.md)–[2](02-ai-agent-processing-and-extraction.md)), through signal scoring ([Page 3](03-signal-scoring-and-weighted-signals.md)) and trend aggregation ([Page 4](04-trend-aggregation-and-accumulating-signals.md)), to the recommendation ([Page 5](05-recommendation-generation.md)), the execution decision, the risk evaluation, and the execution response — every step is persisted and linked by foreign keys. The `execution_decisions` table links to `recommendations` via `recommendation_id`, the `orders` table links back to both, and the `commitments` and `pool_snapshots` tables capture the resource pool impact over time.
+
+For additional reference on the decision execution engine's configuration, queue topology, and database tables, see [docs/services.md](../services.md).
+
+---
+
+## Conclusion: From Raw Data to Decision Execution
+
+This six-page series has traced the full intelligence-to-decision pipeline, from the moment raw data enters the system to the moment an execution request reaches the external execution API.
+
+It began with [Page 1](01-data-ingestion-and-preparation.md), where the scheduler orchestrates ingestion cycles across four data sources — external news, regulatory filings, external data feeds, and macro news APIs — and the parser normalizes raw content into structured documents ready for AI processing. [Page 2](02-ai-agent-processing-and-extraction.md) described how the Document Intelligence Extractor and Global Event Classifier agents use LLM inference to produce structured JSON intelligence, with hot-swappable model configurations and a robust JSON repair pipeline. [Page 3](03-signal-scoring-and-weighted-signals.md) explained how raw extraction output is transformed into `WeightedSignal` objects through a composite formula that balances recency, credibility, novelty, and environmental context across three independent signal layers. [Page 4](04-trend-aggregation-and-accumulating-signals.md) showed how the aggregation engine merges these signals across five time windows, detecting contradictions, ranking evidence, and computing trend projections — with consecutive same-direction signals accumulating to escalate the system's response from neutral through observe and monitor to act or defer. [Page 5](05-recommendation-generation.md) covered the translation of trend assessments into actionable recommendations through data quality suppression, eligibility evaluation, commitment sizing, thesis generation, and risk classification.
+
+And here in Page 6, the pipeline reached its terminus: the decision execution engine's decision loop polling those recommendations, subjecting each to circuit breaker checks, confidence gates, deduplication, pool health assessments, and a multi-step commitment sizer — then submitting approved execution requests through the execution adapter to the external execution API, with every decision recorded in a fully auditable trail from signal to execution.
+
+The pipeline is designed to be conservative by default and transparent throughout. Every stage applies its own safety checks — deduplication at ingestion, confidence gates at extraction, contradiction detection at aggregation, suppression at recommendation, and circuit breakers at execution. The system can be tuned through runtime configuration (risk tier parameters, suppression thresholds, signal layer toggles in `risk_configs`) without code changes or restarts. And the complete audit trail — from `documents` through `document_intelligence`, `document_impact_records`, `trend_windows`, `recommendations`, `execution_decisions`, and `orders` — means that any decision can be traced back to the specific documents, signals, and evaluations that produced it.
diff --git a/docs/sanitized-pipeline-deep-dive/diagrams/.gitkeep b/docs/sanitized-pipeline-deep-dive/diagrams/.gitkeep
new file mode 100644
index 0000000..8b13789
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/diagrams/.gitkeep
@@ -0,0 +1 @@
+
diff --git a/docs/sanitized-pipeline-deep-dive/diagrams/decision-engine-loop.md b/docs/sanitized-pipeline-deep-dive/diagrams/decision-engine-loop.md
new file mode 100644
index 0000000..8f97142
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/diagrams/decision-engine-loop.md
@@ -0,0 +1,94 @@
+# Decision Execution Engine Loop
+
+```mermaid
+flowchart TD
+ subgraph ENGINE["Decision Execution Engine\nservices/trading/engine.py"]
+ direction TB
+ TASKS["5 Concurrent Async Tasks"]
+ T1["_decision_loop()\n60s polling interval"]
+ T2["_risk_threshold_monitor()"]
+ T3["_performance_loop()"]
+ T4["_risk_tier_scheduler()"]
+ T5["_rebalance_scheduler()"]
+ TASKS --> T1 & T2 & T3 & T4 & T5
+ end
+
+ T1 --> POLL["Poll recommendations table\naction IN (act, defer)\nmode IN (simulation_eligible, production_eligible)\ngenerated_at > NOW() − 2h"]
+
+ POLL --> EVAL["evaluate_recommendation()"]
+
+ EVAL --> CHK_A
+
+ subgraph PRETRADE["Pre-Execution Check Sequence\n(first failure short-circuits)"]
+ direction TB
+ CHK_A["a. Circuit Breaker active?\nservices/trading/circuit_breaker.py\nTriggers: daily_loss, single_commitment, volatility"]
+ CHK_B["b. Execution Window?\nis_within_execution_window()"]
+ CHK_C["c. Confidence Gate\nconfidence ≥ risk_tier.min_confidence"]
+ CHK_D["d. Deduplication\nRec ID in processed set?\nRedis: app:dedupe:execution:*"]
+ CHK_E["e. Declining Commitments\n> 50% commitments down > 2%"]
+ CHK_F["f. Max Open Commitments\nopen_count ≥ max (default 10)"]
+
+ CHK_A -->|"pass"| CHK_B
+ CHK_B -->|"pass"| CHK_C
+ CHK_C -->|"pass"| CHK_D
+ CHK_D -->|"pass"| CHK_E
+ CHK_E -->|"pass"| CHK_F
+ end
+
+ CHK_A & CHK_B & CHK_C & CHK_D & CHK_E & CHK_F -->|"fail"| SKIP["ExecutionDecision\ndecision = skip\n+ skip_reason"]
+
+ CHK_F -->|"pass"| SIZER
+
+ subgraph SIZER["Commitment Sizing\nservices/trading/position_sizer.py"]
+ direction TB
+ SZ1["Base sizing\nrisk_tier.max_commitment_pct × 0.5\n× (confidence / min_confidence)"]
+ SZ2["Correlation reduction\nweighted avg corr > 0.8 → reject\n> 0.5 → proportional reduction"]
+ SZ3["Sector exposure\ncap at risk_tier.max_sector_pct"]
+ SZ4["Diversification bonus\n1.2× for new sector (< 3 sectors)"]
+ SZ5["Event proximity\n≤ 1 day → reject\n≤ 3 days → 50% reduction"]
+ SZ6["Absolute commitment cap"]
+ SZ7["Pool exposure check\nmax_pool_exposure × active_pool"]
+ SZ8["Share rounding\nfloor(dollar / price)"]
+
+ SZ1 --> SZ2 --> SZ3 --> SZ4 --> SZ5 --> SZ6 --> SZ7 --> SZ8
+ end
+
+ SIZER -->|"rejected"| SKIP
+ SIZER -->|"approved"| ACT["ExecutionDecision\ndecision = act\nshares, dollar amount"]
+
+ ACT --> PERSIST_TD["Persist to\nexecution_decisions"]
+
+ ACT --> ORDER["Build execution request\n{entity, action, side,\nquantity, request_type}"]
+
+ ORDER -->|"rpush"| Q_BROKER["app:queue:execution_orders"]
+
+ Q_BROKER --> BROKER["Execution Adapter\nexternal execution API (simulation)\nservices/adapters/broker_adapter.py"]
+
+ BROKER --> AUDIT
+
+ subgraph AUDIT["Audit Trail — PostgreSQL"]
+ AU1["execution_requests"]
+ AU2["commitments"]
+ AU3["pool_snapshots"]
+ end
+
+ subgraph CB_DETAIL["Circuit Breaker Detail\nservices/trading/circuit_breaker.py"]
+ CB1["daily_loss\npool loss > 5%\ncooldown: volatility_pause_hours"]
+ CB2["single_commitment\ncommitment loss > 15%\ncooldown: entity_cooldown_hours (48h)"]
+ CB3["volatility\n≥ 3 risk thresholds in 30min\ncooldown: volatility_pause_hours (2h)"]
+ CB4["Redis state\napp:execution:circuit_breaker:*"]
+ end
+
+ subgraph RESERVE["Reserve Pool\nservices/trading/reserve_pool.py"]
+ RP1["Profit siphoning: 20%"]
+ RP2["High-water rebalance: 30%"]
+ RP3["Emergency liquidation"]
+ RP4["reserve_pool_ledger"]
+ end
+
+ subgraph RISK_TIER["Risk Tier Auto-Adjustment\nservices/trading/risk_tier_controller.py"]
+ RT1["Evaluate: risk-adjusted return ratio,\npeak-to-trough decline, success rate"]
+ RT2["conservative → moderate → aggressive"]
+ RT3["risk_tier_history"]
+ end
+```
diff --git a/docs/sanitized-pipeline-deep-dive/diagrams/ingestion-to-extraction-flow.md b/docs/sanitized-pipeline-deep-dive/diagrams/ingestion-to-extraction-flow.md
new file mode 100644
index 0000000..6a0a655
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/diagrams/ingestion-to-extraction-flow.md
@@ -0,0 +1,81 @@
+# Ingestion-to-Extraction Flow
+
+```mermaid
+flowchart TD
+ subgraph Scheduler["Scheduler\nservices/scheduler/app.py"]
+ S1["schedule_cycle()"]
+ S2["Cadence check\nmarket_api: 300s\nnews_api: 300s\nfilings_api: 3600s\nmacro_news: 600s"]
+ S3["Rate limit check\ncheck_rate_limit()"]
+ S1 --> S2 --> S3
+ end
+
+ S3 -->|"rpush"| Q_ING["app:queue:ingestion"]
+
+ Q_ING -->|"lpop"| ING
+
+ subgraph ING["Ingestion Worker\nservices/ingestion/worker.py"]
+ direction TB
+ AD["Adapter Dispatch\nprocess_job()"]
+ AD --> PA["ExternalDataAdapter\nservices/adapters/market_adapter.py"]
+ AD --> PB["ExternalNewsAdapter\nservices/adapters/news_adapter.py"]
+ AD --> PC["RegulatoryFilingsAdapter\nservices/adapters/filings_adapter.py"]
+ AD --> PD["MacroNewsAdapter\nservices/adapters/macro_news_adapter.py"]
+ AD --> PE["WebScrapeAdapter\nservices/adapters/web_scrape_adapter.py"]
+ end
+
+ ING -->|"Content hash check\napp:dedupe:*\nTTL 24h"| REDIS_DEDUPE[("Redis\nDedupe Markers")]
+
+ ING -->|"upload_raw_artifact()"| MINIO_RAW
+
+ subgraph MINIO_RAW["MinIO Raw Storage"]
+ B1["app-raw-data"]
+ B2["app-raw-content"]
+ B3["app-raw-filings"]
+ end
+
+ ING -->|"persist_ingestion_items()"| PG_ING
+
+ subgraph PG_ING["PostgreSQL"]
+ T1["documents"]
+ T2["ingestion_runs"]
+ T3["document_company_mentions"]
+ end
+
+ ING -->|"rpush new doc IDs"| Q_PARSE["app:queue:parsing"]
+
+ Q_PARSE -->|"lpop"| PARSER
+
+ subgraph PARSER["Parser Worker\nservices/parser/worker.py"]
+ P1["fetch_html() → parse_html()"]
+ P2["Quality scoring\nconfidence: high / medium / low"]
+ P3["Company mention detection\ndetect_company_mentions()"]
+ P4["Routing decision"]
+ P1 --> P2 --> P3 --> P4
+ end
+
+ PARSER -->|"upload_normalized_text()\nupload_parser_output()"| MINIO_NORM["MinIO\napp-normalized"]
+ PARSER -->|"update_document_parse_results()"| PG_ING
+
+ P4 -->|"doc_type = macro_event"| Q_MACRO["app:queue:macro_classification"]
+ P4 -->|"doc_type ≠ macro_event"| Q_EXT["app:queue:extraction"]
+
+ Q_EXT -->|"lpop"| EXT
+ Q_MACRO -->|"lpop"| EXT
+
+ subgraph EXT["Extractor Worker\nservices/extractor/main.py"]
+ E1["Document Intelligence\nExtractor agent\nslug: document-extractor"]
+ E2["Global Event Classifier\nslug: event-classifier\nservices/extractor/event_classifier.py"]
+ E3["persist_extraction()\nservices/extractor/worker.py"]
+ end
+
+ EXT -->|"persist to"| PG_EXT
+
+ subgraph PG_EXT["PostgreSQL"]
+ T4["document_intelligence"]
+ T5["document_impact_records"]
+ T6["global_events"]
+ T7["macro_impact_records"]
+ end
+
+ EXT -->|"rpush"| Q_AGG["app:queue:aggregation"]
+```
diff --git a/docs/sanitized-pipeline-deep-dive/diagrams/recommendation-generation-flow.md b/docs/sanitized-pipeline-deep-dive/diagrams/recommendation-generation-flow.md
new file mode 100644
index 0000000..23c9304
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/diagrams/recommendation-generation-flow.md
@@ -0,0 +1,80 @@
+# Recommendation Generation Flow
+
+```mermaid
+flowchart TD
+ Q_REC["app:queue:recommendation"] -->|"lpop"| WORKER["Recommendation Worker\nservices/recommendation/main.py"]
+
+ WORKER --> FETCH["Fetch TrendSummary\nfrom trend_windows\nfor entity + window"]
+
+ FETCH --> SUPP
+
+ subgraph SUPP["Data Quality Suppression\nservices/recommendation/suppression.py"]
+ S1["extraction confidence < 0.40?"]
+ S2["evidence staleness > 168h?"]
+ S3["source diversity < 1 type?"]
+ S4["extraction failure rate > 50%?"]
+ S5["valid documents < 2?"]
+ S6["data quality score < 0.30?"]
+ S7["Macro-only signal?\nevaluate_macro_only_suppression()"]
+ S8["Pattern-only signal?\nevaluate_pattern_only_suppression()"]
+ end
+
+ SUPP -->|"Any check fails:\nsuppressed = true\nmode → informational"| ELIG
+ SUPP -->|"All checks pass"| ELIG
+
+ subgraph ELIG["Eligibility Evaluation\nservices/recommendation/eligibility.py"]
+ direction TB
+ G["Gate Checks"]
+ G1["confidence ≥ 0.35"]
+ G2["strength ≥ 0.10"]
+ G3["contradiction ≤ 0.60"]
+ G4["evidence ≥ 2"]
+ G5["direction ≠ neutral"]
+ G --> G1 & G2 & G3 & G4 & G5
+
+ G1 & G2 & G3 & G4 & G5 --> ACT["Action Mapping"]
+ ACT --> A1["ACT: positive + strength ≥ 0.25"]
+ ACT --> A2["DEFER: negative + strength ≥ 0.25"]
+ ACT --> A3["MONITOR: directional + confidence ≥ 0.50"]
+ ACT --> A4["OBSERVE: otherwise"]
+
+ A1 & A2 & A3 & A4 --> MODE["Mode Escalation"]
+ MODE --> M1["informational\n(default for MONITOR/OBSERVE)"]
+ MODE --> M2["simulation_eligible\nconfidence ≥ 0.50"]
+ MODE --> M3["production_eligible\nconfidence ≥ 0.70\ncontradiction ≤ 0.25\nevidence ≥ 5"]
+ end
+
+ ELIG --> SIZING
+
+ subgraph SIZING["Commitment Sizing\nservices/recommendation/eligibility.py"]
+ PS1["base = 1% allocation pool"]
+ PS2["scale by confidence × strength\nup to 10% max"]
+ PS3["contradiction penalty\n−0.5 × contradiction_score"]
+ PS4["evidence count penalty\n< 3 docs → ×0.5\n< 5 docs → ×0.75"]
+ end
+
+ SIZING --> THESIS
+
+ subgraph THESIS["Thesis Generation"]
+ TH1["Deterministic thesis\nassembled from trend data"]
+ TH2["Optional LLM rewrite\nthesis-rewriter agent\nservices/recommendation/thesis_llm.py"]
+ TH1 --> TH2
+ end
+
+ THESIS --> RISK
+
+ subgraph RISK["Risk Classification"]
+ RC1["low"]
+ RC2["moderate"]
+ RC3["high"]
+ RC4["very_high"]
+ end
+
+ RISK --> PERSIST
+
+ subgraph PERSIST["Persistence — PostgreSQL"]
+ P1["recommendations"]
+ P2["recommendation_evidence"]
+ P3["risk_evaluations"]
+ end
+```
diff --git a/docs/sanitized-pipeline-deep-dive/diagrams/three-layer-signal-merging.md b/docs/sanitized-pipeline-deep-dive/diagrams/three-layer-signal-merging.md
new file mode 100644
index 0000000..901c49b
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/diagrams/three-layer-signal-merging.md
@@ -0,0 +1,52 @@
+# Three-Layer Signal Merging
+
+```mermaid
+flowchart TD
+ subgraph Layer1["Layer 1 — Entity Signals"]
+ DIR["document_impact_records\n(per-entity extraction output)"]
+ DIR -->|"build_weighted_signals()"| WS1["WeightedSignal[]\nweight = 1.0 (full)"]
+ end
+
+ subgraph Layer2["Layer 2 — Macro Signals"]
+ MIR["macro_impact_records\n(global event interpolation)"]
+ MIR -->|"build_macro_weighted_signals()"| WS2["WeightedSignal[]\nimpact × MACRO_SIGNAL_WEIGHT\n(0.3)"]
+ TOGGLE_M{"macro_enabled\nin risk_configs?"}
+ TOGGLE_M -->|"true"| MIR
+ TOGGLE_M -->|"false"| SKIP_M["Layer skipped\ngraceful degradation"]
+ end
+
+ subgraph Layer3["Layer 3 — Competitive Signals"]
+ CSR["competitive_signal_records\n(pattern mining + propagation)"]
+ CSR -->|"build_pattern_weighted_signals()\nservices/aggregation/signal_propagation.py"| WS3["WeightedSignal[]\nimpact × COMPETITIVE_SIGNAL_WEIGHT\n(0.2)"]
+ TOGGLE_C{"competitive_enabled\nin risk_configs?"}
+ TOGGLE_C -->|"true"| CSR
+ TOGGLE_C -->|"false"| SKIP_C["Layer skipped\ngraceful degradation"]
+ end
+
+ WS1 --> MERGE["Concatenate all WeightedSignal lists"]
+ WS2 --> MERGE
+ WS3 --> MERGE
+
+ MERGE --> AGG
+
+ subgraph AGG["Aggregation Engine\nservices/aggregation/worker.py"]
+ A1["weighted_sentiment_average()"]
+ A2["detect_contradictions()\nservices/aggregation/contradiction.py"]
+ A3["derive_trend_direction()"]
+ A4["compute_trend_confidence()"]
+ A5["rank_evidence()"]
+ A1 --> A2 --> A3 --> A4 --> A5
+ end
+
+ AGG -->|"assemble_trend_summary()"| TS["TrendSummary\nservices/shared/schemas.py"]
+
+ TS -->|"persist_trend_summary()"| PG_TREND
+
+ subgraph PG_TREND["PostgreSQL"]
+ TW["trend_windows\n(upserted each cycle)"]
+ TH["trend_history\n(time-series snapshots)"]
+ TE["trend_evidence\n(per-document rankings)"]
+ end
+
+ AGG -->|"rpush"| Q_REC["app:queue:recommendation"]
+```
diff --git a/docs/sanitized-pipeline-deep-dive/diagrams/trend-accumulation-escalation.md b/docs/sanitized-pipeline-deep-dive/diagrams/trend-accumulation-escalation.md
new file mode 100644
index 0000000..ed3dc5e
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/diagrams/trend-accumulation-escalation.md
@@ -0,0 +1,62 @@
+# Trend Accumulation and Escalation
+
+```mermaid
+flowchart TD
+ subgraph Windows["Five Time Windows\nservices/aggregation/worker.py"]
+ W1["intraday (12h)"]
+ W2["1d (1 day)"]
+ W3["7d (7 days)"]
+ W4["30d (30 days)"]
+ W5["90d (90 days)"]
+ end
+
+ W1 & W2 & W3 & W4 & W5 --> SIGNALS
+
+ SIGNALS["Fetch signals per window\nEntity + Macro + Competitive\n→ WeightedSignal[]"]
+
+ SIGNALS --> SENT["weighted_sentiment_average()\nCompute avg sentiment across signals"]
+
+ SENT --> DIR
+
+ subgraph DIR["derive_trend_direction()"]
+ D1["avg_sentiment ≥ 0.15 → POSITIVE"]
+ D2["avg_sentiment ≤ −0.15 → NEGATIVE"]
+ D3["contradiction > 0.10\nAND |avg| < 0.30 → MIXED"]
+ D4["otherwise → NEUTRAL"]
+ end
+
+ DIR --> CONF
+
+ subgraph CONF["compute_trend_confidence()"]
+ C1["Unique source count\ncaps at 15 → 0.8 contribution"]
+ C2["Avg extraction credibility"]
+ C3["Signal agreement ratio\ndampened by log₂(n+1)/log₂(8)\nsaturates ~7 unique sources"]
+ C4["Contradiction penalty\n−0.4 × contradiction_score"]
+ C5["confidence = 0.3×count + 0.3×credibility\n+ 0.4×agreement − penalty"]
+ end
+
+ CONF --> STRENGTH["trend_strength = |avg_sentiment|\nclamped to [0, 1]"]
+
+ STRENGTH --> ESC
+
+ subgraph ESC["Escalation Path\n(via eligibility thresholds)"]
+ direction TB
+ E1["NEUTRAL\nconfidence < 0.35\nOR strength < 0.10\nOR direction = neutral"]
+ E2["OBSERVE\nstrength < 0.25\nAND confidence < 0.50"]
+ E3["MONITOR\nstrength < 0.25\nAND confidence ≥ 0.50"]
+ E4["ACT / DEFER\nstrength ≥ 0.25\nAND direction = positive/negative"]
+
+ E1 -->|"More signals\nsame direction"| E2
+ E2 -->|"Confidence grows\nmore unique sources"| E3
+ E3 -->|"Strength exceeds 0.25\naccumulated evidence"| E4
+ end
+
+ ESC --> PERSIST
+
+ subgraph PERSIST["Persistence"]
+ P1["trend_windows\n(upserted each cycle)"]
+ P2["trend_history\n(time-series snapshots)"]
+ P3["trend_evidence\n(per-document rankings)"]
+ P4["trend_projections\nservices/aggregation/projection.py"]
+ end
+```
diff --git a/docs/sanitized-pipeline-deep-dive/diagrams/weighted-signal-computation.md b/docs/sanitized-pipeline-deep-dive/diagrams/weighted-signal-computation.md
new file mode 100644
index 0000000..17fc9b9
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/diagrams/weighted-signal-computation.md
@@ -0,0 +1,58 @@
+# Weighted Signal Computation
+
+```mermaid
+flowchart TD
+ DOC["Document Signal Input\n(published_at, source_credibility,\nnovelty_score, extraction_confidence,\nmarket_ctx)"]
+
+ DOC --> GATE
+ DOC --> REC
+ DOC --> CRED
+ DOC --> NOV
+ DOC --> MKT
+
+ subgraph GATE["Confidence Gate"]
+ G1["extraction_confidence ≥ 0.2?"]
+ G1 -->|"Yes"| G2["gate = 1.0"]
+ G1 -->|"No"| G3["gate = 0.0\n(signal zeroed out)"]
+ end
+
+ subgraph REC["Recency Decay"]
+ R1["w = 2^(−age_hours / half_life)"]
+ R2["Half-lives per window:\nintraday: 2h\n1d: 12h\n7d: 72h\n30d: 240h\n90d: 720h"]
+ R3["Floor: min_recency_weight = 0.01"]
+ R1 --- R2
+ R1 --- R3
+ end
+
+ subgraph CRED["Source Credibility"]
+ C1["Clamp to [0.1, 1.0]"]
+ C2["Apply exponent\n(default 1.0)"]
+ C1 --> C2
+ end
+
+ subgraph NOV["Novelty Bonus"]
+ N1["bonus = novelty_score × 0.25"]
+ N2["Range: [0.0, 0.25]\n(up to 25% boost)"]
+ N1 --- N2
+ end
+
+ subgraph MKT["Environmental Context Multiplier"]
+ M1["Volatility boost\nlog₁₊(excess) × 0.15\ncapped at 0.30"]
+ M2["Volume surge boost\nvolume_change > 50% → +0.15"]
+ M3["multiplier = 1.0 + boost\n(always ≥ 1.0)"]
+ M1 --> M3
+ M2 --> M3
+ end
+
+ GATE --> FORMULA
+ REC --> FORMULA
+ CRED --> FORMULA
+ NOV --> FORMULA
+ MKT --> FORMULA
+
+ FORMULA["combined = gate × recency × credibility\n× (1 + novelty_bonus)\n× market_context_multiplier"]
+
+ FORMULA --> SW["SignalWeight\nservices/aggregation/scoring.py"]
+
+ SW --> WS["WeightedSignal\n{ document_id, weight: SignalWeight,\nsentiment_value, impact_score }"]
+```
diff --git a/docs/sanitized-pipeline-deep-dive/index.md b/docs/sanitized-pipeline-deep-dive/index.md
new file mode 100644
index 0000000..16ef6dd
--- /dev/null
+++ b/docs/sanitized-pipeline-deep-dive/index.md
@@ -0,0 +1,39 @@
+# Intelligence Pipeline Deep Dive
+
+This document series provides a narrative walkthrough of the full intelligence-to-decision pipeline in the platform. Unlike the existing service reference and API documentation, these pages tell the story of how raw data enters the system, gets processed by AI agents, produces structured signals, accumulates into trend summaries, and ultimately drives autonomous decision execution.
+
+Each page covers one stage of the pipeline and ends with a transition to the next, so you can read the series end-to-end or jump directly to the stage you need. Diagrams are stored as standalone Mermaid files that can be rendered independently or embedded in other documents.
+
+---
+
+## Table of Contents
+
+1. [Data Ingestion and Preparation](01-data-ingestion-and-preparation.md) — How raw data from an external data provider, a public records API, and macro news APIs enters the system, gets deduplicated, stored, parsed, and routed for AI processing.
+2. [AI Agent Processing and Structured Extraction](02-ai-agent-processing-and-extraction.md) — How the Document Intelligence Extractor and Global Event Classifier agents use LLM inference to produce structured JSON intelligence from documents.
+3. [Signal Scoring and the WeightedSignal Abstraction](03-signal-scoring-and-weighted-signals.md) — How raw extraction output is transformed into weighted signals through confidence gating, recency decay, source credibility, novelty bonuses, and environmental context multipliers.
+4. [Trend Aggregation and Accumulating Signals](04-trend-aggregation-and-accumulating-signals.md) — How the aggregation engine merges weighted signals across five time windows, detects contradictions, ranks evidence, and escalates trend strength as consecutive signals accumulate.
+5. [Recommendation Generation](05-recommendation-generation.md) — How trend summaries pass through data quality suppression, eligibility evaluation, commitment sizing, thesis generation, and risk classification to produce actionable recommendations.
+6. [Decision Execution](06-decision-execution.md) — How the decision execution engine polls recommendations, runs pre-execution checks, sizes commitments, enforces circuit breakers, and submits execution requests through the execution adapter.
+
+---
+
+## Diagrams
+
+The following Mermaid diagram files can be rendered independently or referenced from the narrative pages:
+
+- [Ingestion to Extraction Flow](diagrams/ingestion-to-extraction-flow.md) — Flowchart from Scheduler through Ingestion, Parser, to Extractor with all queues and storage.
+- [Three-Layer Signal Merging](diagrams/three-layer-signal-merging.md) — Entity-specific, Environmental, and Relational signal layers converging into the Aggregation engine.
+- [Weighted Signal Computation](diagrams/weighted-signal-computation.md) — Component breakdown of the composite weight formula.
+- [Trend Accumulation and Escalation](diagrams/trend-accumulation-escalation.md) — How consecutive signals strengthen trends and escalate actions across time windows.
+- [Recommendation Generation Flow](diagrams/recommendation-generation-flow.md) — From TrendSummary through suppression, eligibility, thesis, risk classification, to persistence.
+- [Decision Engine Loop](diagrams/decision-engine-loop.md) — Pre-execution check sequence, commitment sizing, and execution request submission flow.
+
+---
+
+## Related Documentation
+
+For reference-level detail on individual services, AI agent configuration, and infrastructure, see the existing documentation:
+
+- [Services Reference](../services.md) — Per-service configuration, database tables, queues, and runtime behaviors.
+- [AI Agents Guide](../ai-agents.md) — AI agent configuration, variants, A/B testing, and the agent management API.
+- [Data Pipeline Architecture](../architecture-data-pipeline.md) — Queue topology, data store summary, and Mermaid flow diagrams for the full data pipeline.
diff --git a/docs/services.md b/docs/services.md
new file mode 100644
index 0000000..372f89b
--- /dev/null
+++ b/docs/services.md
@@ -0,0 +1,1052 @@
+# Service Documentation
+
+Stonks Oracle is composed of 13 services that form an end-to-end AI market intelligence and paper-trading pipeline. This document describes each service's purpose, entry point, configuration, database tables, Redis queue interactions, and key features.
+
+For services that expose HTTP endpoints, see the [API Reference](api-reference.md).
+
+---
+
+## Table of Contents
+
+1. [Queue Topology](#queue-topology)
+2. [Scheduler](#1-scheduler)
+3. [Symbol Registry](#2-symbol-registry)
+4. [Ingestion](#3-ingestion)
+5. [Parser](#4-parser)
+6. [Extractor](#5-extractor)
+7. [Aggregation](#6-aggregation)
+8. [Recommendation](#7-recommendation)
+9. [Trading Engine](#8-trading-engine)
+10. [Risk Engine](#9-risk-engine)
+11. [Broker Adapter](#10-broker-adapter)
+12. [Lake Publisher](#11-lake-publisher)
+13. [Query API](#12-query-api)
+14. [Dashboard](#13-dashboard)
+15. [Signal Layers](#signal-layers)
+16. [Trading Engine Features](#trading-engine-features)
+
+---
+
+## Queue Topology
+
+All queues use the `stonks:queue:<name>` key pattern (configurable via `DEPLOY_STAGE` prefix). Dead-letter queues follow the pattern `stonks:dlq:<name>`.
+
+| Queue Name | Full Redis Key | Producer(s) | Consumer |
+|---|---|---|---|
+| `ingestion` | `stonks:queue:ingestion` | Scheduler | Ingestion |
+| `parsing` | `stonks:queue:parsing` | Ingestion | Parser |
+| `extraction` | `stonks:queue:extraction` | Parser, Scheduler (recovery) | Extractor |
+| `macro_classification` | `stonks:queue:macro_classification` | Parser, Scheduler (recovery) | Extractor |
+| `aggregation` | `stonks:queue:aggregation` | Extractor | Aggregation |
+| `recommendation` | `stonks:queue:recommendation` | Aggregation | Recommendation |
+| `broker_orders` | `stonks:queue:broker_orders` | Trading Engine, Trading API | Broker Adapter |
+| `lake_publish` | `stonks:queue:lake_publish` | Various services | Lake Publisher |
+
+### Queue Message Schemas
+
+**Ingestion Job** (`stonks:queue:ingestion`):
+```json
+{
+ "source_id": "uuid",
+ "company_id": "uuid | null",
+ "ticker": "AAPL",
+ "legal_name": "Apple Inc.",
+ "aliases": ["Apple", "AAPL"],
+ "source_type": "news_api",
+ "source_name": "Polygon News",
+ "config": {},
+ "credibility_score": 0.5,
+ "scheduled_at": "2025-01-01T00:00:00+00:00"
+}
+```
+
+**Parsing Job** (`stonks:queue:parsing`):
+```json
+{
+ "document_id": "uuid",
+ "ticker": "AAPL",
+ "source_type": "news_api"
+}
+```
+
+**Extraction Job** (`stonks:queue:extraction`):
+```json
+{
+ "document_id": "uuid",
+ "ticker": "AAPL",
+ "normalized_text": "Article text content..."
+}
+```
+
+**Macro Classification Job** (`stonks:queue:macro_classification`):
+```json
+{
+ "document_id": "uuid",
+ "ticker": "",
+ "normalized_text": "Global event text..."
+}
+```
+
+**Aggregation Job** (`stonks:queue:aggregation`):
+```json
+{
+ "ticker": "AAPL",
+ "macro_event_id": "uuid (optional)"
+}
+```
+
+**Recommendation Job** (`stonks:queue:recommendation`):
+```json
+{
+ "ticker": "AAPL",
+ "window": "7d"
+}
+```
+
+**Broker Order Job** (`stonks:queue:broker_orders`):
+```json
+{
+ "ticker": "AAPL",
+ "side": "buy",
+ "quantity": 10.0,
+ "order_type": "market",
+ "limit_price": null,
+ "stop_price": null,
+ "recommendation_id": "uuid",
+ "confidence": 0.75,
+ "estimated_value": 1500.0,
+ "sector": "Technology",
+ "source": "trading_engine",
+ "idempotency_key": "optional-explicit-key"
+}
+```
+
+**Lake Publish Job** (`stonks:queue:lake_publish`):
+```json
+{
+ "job_type": "document | document_extraction | market_snapshot | trade_order | trade_fill | positions_snapshot | pnl_snapshot | global_event | macro_impact | trend_projection | competitor_relationship | competitive_signal | bulk_documents | bulk_extractions",
+ "entity_id": "uuid or ticker",
+ "dt": "2025-01-01T00:00:00+00:00",
+ "since": "2025-01-01T00:00:00+00:00 (for bulk jobs)"
+}
+```
+
+---
+
+## 1. Scheduler
+
+**Purpose**: Triggers ingestion cycles for tracked companies and sources on a configurable cadence. Polls the symbol registry for active companies and their configured sources, respects per-source polling intervals and backoff windows, coordinates rate limits across source types, and enqueues ingestion jobs for downstream workers. Also runs periodic maintenance: stale document recovery, failed extraction retries, and data retention cleanup.
+
+**Entry Point**: `services.scheduler.app`
+
+**Tier**: Orchestration (no HTTP endpoints)
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_HOST` | `localhost` | PostgreSQL host |
+| `POSTGRES_PORT` | `5432` | PostgreSQL port |
+| `POSTGRES_DB` | `stonks` | Database name |
+| `POSTGRES_USER` | `stonks` | Database user |
+| `POSTGRES_PASSWORD` | `stonks_dev` | Database password |
+| `REDIS_HOST` | `localhost` | Redis host |
+| `REDIS_PORT` | `6379` | Redis port |
+| `REDIS_PASSWORD` | _(none)_ | Redis password |
+| `LOG_LEVEL` | `INFO` | Logging level |
+| `JSON_LOGS` | `true` | Structured JSON logging |
+| `PIPELINE_DEFAULT_OFF` | _(none)_ | If `true`, initializes the pipeline toggle to OFF on first boot |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `sources` | Read | Active sources with polling config |
+| `companies` | Read | Active companies and tickers |
+| `company_aliases` | Read | Aliases for entity matching |
+| `ingestion_runs` | Read | Last run status, timing, retry state |
+| `documents` | Read/Write | Stale document recovery, status reset |
+| `document_intelligence` | Write (delete) | Clear failed extractions for retry |
+| `competitive_signal_records` | Write (delete) | Retention cleanup |
+| `trading_decisions` | Write (delete) | Retention cleanup |
+| `risk_evaluations` | Write (delete) | Retention cleanup |
+| `audit_events` | Write (delete) | Retention cleanup |
+| `macro_impact_records` | Write (delete) | Retention cleanup |
+| `recommendation_evidence` | Write (delete) | Retention cleanup |
+| `recommendations` | Write (delete) | Retention cleanup |
+| `order_events` | Write (delete) | Retention cleanup |
+| `model_performance_metrics` | Write (delete) | Retention cleanup |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Publish | `stonks:queue:ingestion` | Enqueue ingestion jobs for due sources |
+| Read | `stonks:pipeline:enabled` | Pipeline toggle (skip cycle if `"0"`) |
+| Read/Write | `stonks:lock:scheduler_cycle` | Distributed lock for single-writer |
+| Read/Write | `stonks:ratelimit:*` | Per-source-type and global Polygon rate limits |
+| Read/Write | `stonks:queue:enqueued:*` | Dedup markers for recovery re-enqueue |
+
+### Key Behaviors
+
+- **Polling cadences**: Default intervals per source type — `market_api`: 300s, `news_api`: 300s, `filings_api`: 3600s, `web_scrape`: 1800s, `broker`: 30s, `macro_news`: 600s. Overridable per-source via `config.polling_interval_seconds`.
+- **Rate limiting**: Per-type limits (e.g., `market_api`: 20/min) plus a global Polygon limit of 45/min across `market_api` + `news_api`.
+- **Backoff**: Exponential backoff on failures (base 60s, max 3600s, max 10 retries).
+- **Stale document recovery**: Every ~5 minutes, re-enqueues documents stuck in `parsed` status for >240 minutes.
+- **Failed extraction retry**: Every ~10 minutes, re-enqueues `extraction_failed` documents older than 60 minutes.
+- **Data retention cleanup**: Every ~25 minutes, deletes old rows from 10 tables with configurable retention windows (14–90 days).
+
+---
+
+## 2. Symbol Registry
+
+**Purpose**: Manages the tracked universe of companies, their aliases, watchlists, data sources, exposure profiles, and competitor relationships. Provides a CRUD API for the symbol registry used by all other services.
+
+**Entry Point**: `services.symbol_registry.app` (FastAPI)
+
+**Tier**: API (HTTP endpoints, see [API Reference](api-reference.md))
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_HOST` | `localhost` | PostgreSQL host |
+| `POSTGRES_PORT` | `5432` | PostgreSQL port |
+| `POSTGRES_DB` | `stonks` | Database name |
+| `POSTGRES_USER` | `stonks` | Database user |
+| `POSTGRES_PASSWORD` | `stonks_dev` | Database password |
+| `LOG_LEVEL` | `INFO` | Logging level |
+| `JSON_LOGS` | `true` | Structured JSON logging |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `companies` | CRUD | Tracked companies (ticker, legal name, sector, industry) |
+| `company_aliases` | CRUD | Alternative names for entity matching |
+| `watchlists` | CRUD | Named groups of companies |
+| `watchlist_members` | CRUD | Company membership in watchlists |
+| `sources` | CRUD | Data source configurations per company |
+| `exposure_profiles` | CRUD | Geographic/commodity exposure for macro interpolation |
+| `competitor_relationships` | CRUD | Competitor pairs with relationship type and strength |
+
+### Redis Queues
+
+None — this service is purely HTTP-driven.
+
+---
+
+## 3. Ingestion
+
+**Purpose**: Fetches raw data from external sources (Polygon market data, Polygon news, SEC EDGAR filings, web scraping, Alpaca broker, macro news). Stores raw payloads in MinIO, deduplicates content, persists document metadata to PostgreSQL, and enqueues new documents for parsing.
+
+**Entry Point**: `services.ingestion.worker`
+
+**Tier**: Pipeline (queue-driven worker)
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `MINIO_ENDPOINT` | `localhost:9000` | MinIO endpoint |
+| `MINIO_ACCESS_KEY` | `minioadmin` | MinIO access key |
+| `MINIO_SECRET_KEY` | `minioadmin` | MinIO secret key |
+| `MARKET_DATA_API_KEY` | _(empty)_ | Polygon.io API key |
+| `MARKET_DATA_BASE_URL` | `https://api.polygon.io` | Polygon base URL |
+| `BROKER_API_KEY` | _(none)_ | Alpaca API key |
+| `BROKER_API_SECRET` | _(none)_ | Alpaca API secret |
+| `BROKER_BASE_URL` | _(none)_ | Alpaca base URL |
+| `BROKER_MODE` | `paper` | Broker mode (`paper` or `live`) |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `ingestion_runs` | Write | Record each ingestion attempt with status |
+| `documents` | Write | Persist document metadata |
+| `document_company_mentions` | Write | Link documents to mentioned companies |
+| `sources` | Write | Update `last_published_at` in source config |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Consume | `stonks:queue:ingestion` | Receive ingestion jobs from scheduler |
+| Publish | `stonks:queue:parsing` | Enqueue parsed documents for parsing |
+| Read/Write | `stonks:dedupe:*` | Content hash deduplication markers (24h TTL) |
+
+### MinIO Buckets
+
+- `stonks-raw-market` — Raw market data JSON
+- `stonks-raw-news` — Raw news article JSON
+- `stonks-raw-filings` — Raw SEC filing data
+- `stonks-normalized` — Normalized text (written by parser)
+
+### Adapters
+
+| Source Type | Adapter Class | External API |
+|---|---|---|
+| `market_api` | `PolygonMarketAdapter` | Polygon.io |
+| `news_api` | `PolygonNewsAdapter` | Polygon.io |
+| `filings_api` | `SECEdgarAdapter` | SEC EDGAR |
+| `web_scrape` | `WebScrapeAdapter` | Direct HTTP |
+| `broker` | `AlpacaBrokerAdapter` | Alpaca |
+| `macro_news` | `MacroNewsAdapter` | Polygon.io |
+
+---
+
+## 4. Parser
+
+**Purpose**: Converts raw HTML/text into normalized, quality-scored documents. Uses BeautifulSoup for HTML parsing, extracts metadata (title, author, publisher, canonical URL), detects company mentions via alias matching, computes quality scores, and routes documents to the appropriate extraction queue.
+
+**Entry Point**: `services.parser.worker`
+
+**Tier**: Pipeline (queue-driven worker)
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `MINIO_*` | _(see shared)_ | MinIO connection |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `documents` | Read/Write | Fetch URL, update parse results and status |
+| `company_aliases` | Read | Alias lookup for company mention detection |
+| `companies` | Read | Ticker and legal name for mention detection |
+| `document_company_mentions` | Write | Persist detected company mentions |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Consume | `stonks:queue:parsing` | Receive documents from ingestion |
+| Publish | `stonks:queue:extraction` | Enqueue standard documents for LLM extraction |
+| Publish | `stonks:queue:macro_classification` | Route `macro_event` documents to event classifier |
+
+### MinIO Buckets
+
+- `stonks-normalized` — Normalized text output
+- `stonks-audit` — Structured parser output JSON (metadata, quality signals, mentions)
+
+### Key Behaviors
+
+- Fetches article HTML via `httpx` if a URL is available
+- Enriches short articles (<500 chars) with Polygon description from raw payload
+- Quality scoring with confidence levels (`high`, `medium`, `low`)
+- Low-quality documents (`confidence = "low"`) are marked but not sent for extraction
+- Routes `macro_event` documents to the macro classification queue instead of standard extraction
+
+---
+
+## 5. Extractor
+
+**Purpose**: Performs LLM-based intelligence extraction from documents using Ollama. Handles two pipelines: (1) standard document extraction producing `DocumentIntelligence` with per-company impact records, and (2) macro event classification producing `GlobalEventSchema` with company-level macro impact interpolation. Supports AI agent configuration with variant-based A/B testing.
+
+**Entry Point**: `services.extractor.main`
+
+**Tier**: Pipeline (queue-driven worker)
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `MINIO_*` | _(see shared)_ | MinIO connection |
+| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama API endpoint |
+| `OLLAMA_MODEL` | `qwen3.5:9b` | Default LLM model |
+| `OLLAMA_TIMEOUT` | `120` | Request timeout (seconds) |
+| `OLLAMA_MAX_RETRIES` | `2` | Max retry attempts |
+| `MACRO_CONFIDENCE_THRESHOLD` | `0.4` | Minimum confidence for macro event inclusion |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `documents` | Read/Write | Fetch document type, normalized text ref; update status |
+| `document_intelligence` | Write | Persist extraction results |
+| `document_impact_records` | Write | Per-company impact scores |
+| `companies` | Read | Ticker-to-company-id mapping |
+| `global_events` | Write | Persist classified macro events |
+| `macro_impact_records` | Write | Per-company macro impact scores |
+| `exposure_profiles` | Read | Company exposure profiles for interpolation |
+| `ai_agents` | Read | Agent configuration (model, prompts) |
+| `agent_variants` | Read | Active variant overrides for A/B testing |
+| `agent_performance_log` | Write | Performance metrics per extraction |
+| `model_performance_metrics` | Write | Extraction quality metrics |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Consume | `stonks:queue:extraction` | Standard document extraction jobs |
+| Consume | `stonks:queue:macro_classification` | Macro event classification jobs |
+| Publish | `stonks:queue:aggregation` | Trigger aggregation after extraction |
+
+### Key Behaviors
+
+- Alternates between macro and extraction queues (1 macro per 3 jobs) to prevent starvation
+- Resolves agent configuration from DB with 60-second TTL cache (`AgentConfigResolver`)
+- Supports separate models for document extraction and event classification
+- Token budget enforcement per variant (hourly limit)
+- Input token limit truncation (configurable per variant)
+- Refreshes company map and agent config every 100 jobs
+- Consecutive macro classification failures trigger operator alerts (threshold: 3)
+
+---
+
+## 6. Aggregation
+
+**Purpose**: Computes rolling-window trend summaries for each company by merging signals from three layers: company-specific document intelligence, macro impact records, and competitive signal propagation. Produces `TrendSummary` objects with direction, strength, confidence, evidence rankings, and contradiction analysis.
+
+**Entry Point**: `services.aggregation.main`
+
+**Tier**: Pipeline (queue-driven worker)
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `MACRO_SIGNAL_WEIGHT` | `0.3` | Relative weight of macro signals |
+| `MACRO_ENABLED` | `true` | Enable/disable macro signal layer |
+| `COMPETITIVE_SIGNAL_WEIGHT` | `0.2` | Relative weight of competitive signals |
+| `COMPETITIVE_ENABLED` | `true` | Enable/disable competitive signal layer |
+| `COMPETITIVE_PATTERN_CONFIDENCE_THRESHOLD` | `0.3` | Minimum pattern confidence |
+| `COMPETITIVE_PROPAGATION_STRENGTH_THRESHOLD` | `0.2` | Minimum propagation strength |
+| `COMPETITIVE_PROPAGATION_FAILURE_THRESHOLD` | `5` | Consecutive failures before alert |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `document_impact_records` | Read | Company-specific signals from extraction |
+| `document_intelligence` | Read | Extraction metadata and confidence |
+| `documents` | Read | Document publication dates and status |
+| `macro_impact_records` | Read | Macro impact scores per company |
+| `global_events` | Read | Source document for macro events |
+| `competitive_signal_records` | Read/Write | Competitive signals targeting a ticker |
+| `competitor_relationships` | Read | Competitor pairs for signal propagation |
+| `trend_windows` | Write (upsert) | Current trend summaries per ticker/window |
+| `trend_history` | Write | Time-series snapshots for charting |
+| `trend_evidence` | Write | Detailed evidence rankings per trend |
+| `trend_projections` | Write | Forward-looking trend projections |
+| `risk_configs` | Read | Runtime toggle state for macro/competitive layers |
+| `market_snapshots` | Read | Market context (price, volume, volatility) |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Consume | `stonks:queue:aggregation` | Receive aggregation jobs from extractor |
+| Publish | `stonks:queue:recommendation` | Enqueue recommendation jobs per ticker/window |
+| Read/Write | `stonks:rec_dedup:*` | Dedup markers for recommendation queue (5-min TTL) |
+
+### Key Behaviors
+
+- Computes trend summaries across 5 windows: `intraday`, `1d`, `7d`, `30d`, `90d`
+- Merges three signal layers via the `WeightedSignal` abstraction (see [Signal Layers](#signal-layers))
+- Triggers competitive signal propagation after aggregation when the competitive layer is enabled
+- Reads toggle state from `risk_configs` table on each cycle (no restart needed)
+- Contradiction detection identifies disagreements between signals
+- Evidence ranking uses composite scoring (weight, impact, recency, confidence)
+
+---
+
+## 7. Recommendation
+
+**Purpose**: Generates actionable trading recommendations from trend summaries. Builds a thesis with supporting evidence, classifies risk level, determines action (buy/sell/hold/watch) and mode (informational/paper_eligible/live_eligible), and optionally rewrites the thesis using an LLM.
+
+**Entry Point**: `services.recommendation.main`
+
+**Tier**: Pipeline (queue-driven worker)
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `MINIO_*` | _(see shared)_ | MinIO connection |
+| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama API (for thesis rewriting) |
+| `OLLAMA_MODEL` | `qwen3.5:9b` | Default model |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `trend_windows` | Read | Latest trend summaries per ticker/window |
+| `trend_projections` | Read | Forward-looking projections |
+| `recommendations` | Write | Persist generated recommendations |
+| `recommendation_evidence` | Write | Link recommendations to source documents |
+| `ai_agents` | Read | Thesis rewriter agent config |
+| `agent_variants` | Read | Active variant for thesis rewriter |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Consume | `stonks:queue:recommendation` | Receive recommendation jobs from aggregation |
+
+### Key Behaviors
+
+- Resolves `thesis-rewriter` agent config from DB (60-second TTL cache)
+- Refreshes agent config every 50 jobs
+- Deduplicates recommendations to avoid generating identical entries
+- Classifies recommendations into risk tiers based on confidence and signal strength
+- Mode classification: `informational` (low confidence), `paper_eligible` (medium), `live_eligible` (high)
+- Pattern-only and macro-only trend shifts are forced to `informational` mode (suppression safety)
+
+---
+
+## 8. Trading Engine
+
+**Purpose**: Autonomous trading engine that polls for new recommendations, evaluates position sizing, manages circuit breakers and reserve pools, auto-adjusts risk tiers, runs backtests, and sends notifications. Exposes an HTTP API for engine control, decision audit, performance metrics, backtesting, and manual override orders.
+
+**Entry Point**: `services.trading.app` (FastAPI)
+
+**Tier**: Trading (HTTP endpoints, see [API Reference](api-reference.md))
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `TRADING_ENABLED` | `false` | Enable autonomous trading |
+| `TRADING_RISK_TIER` | `moderate` | Risk tier (`conservative`, `moderate`, `aggressive`) |
+| `TRADING_RESERVE_SIPHON_PCT` | `0.20` | Percentage of profits siphoned to reserve pool |
+| `TRADING_POLLING_INTERVAL_SECONDS` | `60` | Recommendation polling interval |
+| `TRADING_STOP_LOSS_CHECK_INTERVAL_SECONDS` | `300` | Stop-loss monitoring interval |
+| `TRADING_FAST_STOP_LOSS_INTERVAL_SECONDS` | `60` | Fast stop-loss check interval |
+| `TRADING_GRADUAL_ENTRY_TRANCHES` | `3` | Number of tranches for gradual entry |
+| `TRADING_GRADUAL_ENTRY_THRESHOLD_DOLLARS` | `30.0` | Minimum order value for gradual entry |
+| `TRADING_ABSOLUTE_POSITION_CAP` | `50.0` | Maximum position size (dollars) |
+| `TRADING_ACTIVE_POOL_MINIMUM` | `100.0` | Minimum active pool balance |
+| `TRADING_EMERGENCY_DRAWDOWN_THRESHOLD_PCT` | `0.40` | Emergency drawdown threshold |
+| `TRADING_RESERVE_HIGH_WATER_PCT` | `0.30` | Reserve pool high-water mark |
+| `TRADING_MICRO_TRADING_ENABLED` | `false` | Enable micro-trading mode |
+| `TRADING_MICRO_TRADING_INTERVAL_SECONDS` | `300` | Micro-trading polling interval |
+| `TRADING_MICRO_TRADING_ALLOCATION_CAP_PCT` | `0.03` | Max allocation per micro-trade |
+| `TRADING_MICRO_TRADING_MAX_DAILY` | `10` | Max micro-trades per day |
+| `TRADING_MICRO_TRADING_MAX_HOLD_MINUTES` | `120` | Max hold time for micro-trades |
+| `TRADING_MAX_OPEN_POSITIONS` | `10` | Maximum concurrent open positions |
+| `TRADING_SNS_TOPIC_ARN` | _(empty)_ | AWS SNS topic ARN for SMS notifications |
+| `TRADING_SNS_PHONE_NUMBER` | _(empty)_ | Phone number for SMS notifications |
+| `TRADING_GMAIL_SENDER` | _(empty)_ | Gmail sender address |
+| `TRADING_GMAIL_RECIPIENT` | _(empty)_ | Gmail recipient address |
+| `BROKER_API_KEY` | _(none)_ | Alpaca API key |
+| `BROKER_API_SECRET` | _(none)_ | Alpaca API secret |
+| `BROKER_BASE_URL` | _(none)_ | Alpaca base URL |
+| `SYMBOL_REGISTRY_URL` | `http://symbol-registry:8000` | Symbol registry service URL |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `recommendations` | Read | Poll for new actionable recommendations |
+| `trading_decisions` | Write | Persist every decision (act/skip) with full trace |
+| `portfolio_snapshots` | Read/Write | Daily portfolio performance snapshots |
+| `reserve_pool_ledger` | Read/Write | Reserve pool transactions |
+| `risk_tier_history` | Read/Write | Risk tier change audit trail |
+| `circuit_breaker_events` | Read/Write | Circuit breaker trigger/reset events |
+| `positions` | Read | Current open positions |
+| `position_stop_levels` | Read/Write | Stop-loss and take-profit levels |
+| `orders` | Read | Order history for dedup |
+| `backtest_runs` | Read/Write | Backtest configuration and results |
+| `backtest_trades` | Read/Write | Individual trades within a backtest |
+| `notifications` | Write | Notification history |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Publish | `stonks:queue:broker_orders` | Submit orders to broker adapter |
+| Read/Write | `stonks:dedupe:trading:*` | Recommendation dedup markers |
+| Read/Write | `stonks:trading:circuit_breaker:*` | Circuit breaker state |
+| Read/Write | `stonks:trading:notification_rate:*` | Notification rate limiting |
+
+See [Trading Engine Features](#trading-engine-features) for detailed feature documentation.
+
+---
+
+## 9. Risk Engine
+
+**Purpose**: Evaluates proposed orders against portfolio risk rules and manages an operator approval workflow. Provides an HTTP API for order evaluation, pending approval listing, approval review, and expiration of stale approvals.
+
+**Entry Point**: `services.risk.app` (FastAPI)
+
+**Tier**: Trading (HTTP endpoints, see [API Reference](api-reference.md))
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `risk_configs` | Read | Active risk configuration |
+| `risk_evaluations` | Write | Persist evaluation results |
+| `approval_requests` | Read/Write | Operator approval workflow |
+| `daily_risk_snapshots` | Read | Daily portfolio risk state |
+
+### Redis Queues
+
+None — called synchronously by the broker adapter and via HTTP.
+
+### Key Behaviors
+
+- Evaluates orders against configurable risk checks (position limits, sector concentration, daily loss limits, portfolio heat)
+- Returns `eligible` / `not eligible` with detailed check results and rejection reasons
+- Approval workflow: orders requiring approval are held until an operator reviews them
+- Stale approvals can be expired via the `/approvals/expire` endpoint
+
+---
+
+## 10. Broker Adapter
+
+**Purpose**: Processes order requests from the broker queue, evaluates them through the risk engine, submits to Alpaca's paper trading API, and persists the full audit trail. Implements idempotent order submission with Redis fast-path and PostgreSQL durable fallback duplicate detection. Periodically syncs positions and order statuses from Alpaca.
+
+**Entry Point**: `services.adapters.broker_service`
+
+**Tier**: Trading (queue-driven worker)
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `MINIO_*` | _(see shared)_ | MinIO connection (for lake publishing) |
+| `BROKER_API_KEY` | _(none)_ | Alpaca API key |
+| `BROKER_API_SECRET` | _(none)_ | Alpaca API secret |
+| `BROKER_BASE_URL` | _(none)_ | Alpaca base URL |
+| `BROKER_MODE` | `paper` | Trading mode (`paper` or `live`) |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `orders` | Write | Persist submitted/rejected/filled orders |
+| `order_events` | Write | Order lifecycle events (submitted, fill, rejected) |
+| `risk_evaluations` | Write | Risk evaluation results per order |
+| `positions` | Write (upsert) | Sync positions from Alpaca |
+| `broker_accounts` | Write (upsert) | Register/update broker account |
+| `daily_risk_snapshots` | Read | Daily portfolio state for risk evaluation |
+| `risk_configs` | Read | Active risk configuration |
+| `approval_requests` | Write | Create approval requests for gated orders |
+| `audit_events` | Write | Full audit trail |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Consume | `stonks:queue:broker_orders` | Receive order jobs from trading engine |
+| Read/Write | `stonks:order_idempotency:*` | Idempotency markers (24h TTL) |
+
+### Key Behaviors
+
+- **Idempotent submission**: Deterministic key generation from job attributes; Redis fast-path + PostgreSQL durable fallback
+- **Risk evaluation**: Every order is evaluated through the risk engine before submission
+- **Approval gate**: Orders requiring operator approval are held (not submitted) until reviewed
+- **Position sync**: Every 60 seconds, syncs positions and order statuses from Alpaca
+- **Lake publishing**: Publishes order facts, fill facts, and position snapshots to the analytical lake
+- **Audit trail**: Records every step (risk evaluation, submission, fill, rejection, duplicate prevention)
+
+---
+
+## 11. Lake Publisher
+
+**Purpose**: Transforms operational data from PostgreSQL into analytical Parquet fact tables stored in MinIO. Supports 15 job types covering documents, extractions, market data, orders, fills, positions, PnL, global events, macro impacts, trend projections, competitor relationships, and competitive signals. Data is queryable via Trino and visualized in Superset and the React dashboard.
+
+**Entry Point**: `services.lake_publisher.jobs`
+
+**Tier**: Analytics (queue-driven worker)
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `MINIO_*` | _(see shared)_ | MinIO connection |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+| Table | Access | Purpose |
+|---|---|---|
+| `documents` | Read | Document metadata for fact publishing |
+| `document_intelligence` | Read | Extraction results |
+| `document_impact_records` | Read | Per-company impact scores |
+| `market_snapshots` | Read | Market bar/quote data |
+| `orders` | Read | Trade orders |
+| `order_events` | Read | Fill events |
+| `positions` | Read | Current positions |
+| `broker_accounts` | Read | Broker account metadata |
+| `global_events` | Read | Macro event classifications |
+| `macro_impact_records` | Read | Macro impact scores |
+| `trend_projections` | Read | Trend projections |
+| `competitor_relationships` | Read | Competitor pairs |
+| `competitive_signal_records` | Read | Competitive signals |
+| `companies` | Read | Company names for enrichment |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Consume | `stonks:queue:lake_publish` | Receive lake publish jobs |
+
+### MinIO Buckets
+
+- `stonks-lakehouse` — Partitioned Parquet fact tables
+
+### Supported Job Types
+
+`document`, `document_extraction`, `market_snapshot`, `trade_order`, `trade_fill`, `positions_snapshot`, `pnl_snapshot`, `global_event`, `macro_impact`, `trend_projection`, `competitor_relationship`, `competitive_signal`, `bulk_documents`, `bulk_extractions`
+
+---
+
+## 12. Query API
+
+**Purpose**: Read-only FastAPI service for analytics, evidence drill-down, and admin controls. Serves the React dashboard and external integrations with endpoints for companies, documents, trends, recommendations, orders, positions, portfolio metrics, global events, macro impacts, competitive signals, trend projections, AI agents, dead-letter queues, pipeline control, SQL explorer, saved queries, audit trail, DevOps metrics, and Prometheus metrics.
+
+**Entry Point**: `services.api.app` (FastAPI)
+
+**Tier**: API (HTTP endpoints, see [API Reference](api-reference.md))
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_*` | _(see shared)_ | PostgreSQL connection |
+| `REDIS_*` | _(see shared)_ | Redis connection |
+| `MINIO_*` | _(see shared)_ | MinIO connection |
+| `TRINO_HOST` | `localhost` | Trino host |
+| `TRINO_PORT` | `8080` | Trino port |
+| `TRINO_CATALOG` | `lakehouse` | Trino catalog |
+| `TRINO_SCHEMA` | `stonks` | Trino schema |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+### Database Tables
+
+The Query API reads from nearly all tables in the database, including:
+
+| Table | Purpose |
+|---|---|
+| `companies`, `company_aliases` | Company listings and details |
+| `sources` | Source configurations |
+| `documents`, `document_company_mentions` | Document timelines |
+| `document_intelligence`, `document_impact_records` | Intelligence extraction results |
+| `trend_windows`, `trend_history`, `trend_projections` | Trend summaries and projections |
+| `recommendations`, `recommendation_evidence` | Recommendation history with evidence |
+| `risk_evaluations` | Risk evaluation results |
+| `orders`, `order_events` | Order history and lifecycle |
+| `positions`, `portfolio_snapshots` | Portfolio state |
+| `global_events`, `macro_impact_records` | Macro event data |
+| `competitive_signal_records`, `competitor_relationships` | Competitive intelligence |
+| `ai_agents`, `agent_variants`, `agent_performance_log` | AI agent management |
+| `audit_events` | Audit trail |
+| `market_snapshots` | Market price data |
+| `watchlists`, `watchlist_members` | Watchlist data |
+
+### Redis Queues
+
+| Direction | Queue | Purpose |
+|---|---|---|
+| Read/Write | `stonks:pipeline:enabled` | Pipeline toggle control |
+| Read | `stonks:queue:*` | Queue depth monitoring for DLQ and DevOps metrics |
+| Read | `stonks:dlq:*` | Dead-letter queue inspection and replay |
+
+### Key Behaviors
+
+- Exposes `/metrics` endpoint for Prometheus scraping
+- Trace context propagation via `x-trace-id` header middleware
+- SQL explorer endpoint for ad-hoc Trino queries
+- Dead-letter queue management (list, inspect, replay)
+- Pipeline control (enable/disable via Redis toggle)
+- Saved queries with CRUD operations
+
+---
+
+## 13. Dashboard
+
+**Purpose**: React frontend serving the web dashboard via nginx. Provides a visual interface for monitoring the platform: company overviews, document timelines, trend charts, recommendation cards, trading performance, portfolio state, and administrative controls.
+
+**Entry Point**: `frontend/Dockerfile` (nginx on port 8080)
+
+**Tier**: Frontend
+
+### Configuration
+
+The dashboard is a static React build served by nginx. It proxies API requests:
+
+| Proxy Path | Backend Service |
+|---|---|
+| `/api/` | Query API (port 8000) |
+| `/registry/` | Symbol Registry (port 8000) |
+| `/risk/` | Risk Engine (port 8000) |
+
+### Technology Stack
+
+- React 19, TypeScript (strict mode)
+- Tailwind CSS for styling
+- TanStack Router and TanStack Query for routing and data fetching
+- Recharts for data visualization
+- MSW (Mock Service Worker) for testing
+
+---
+
+## Signal Layers
+
+The aggregation engine merges three independent signal layers into a unified `WeightedSignal` abstraction. Each layer can be toggled at runtime via the `risk_configs` table (no service restart required).
+
+### Layer 1: Company-Specific Signals
+
+**Data flow**: Documents → Ingestion → Parsing → Extraction → `document_impact_records` → `WeightedSignal` → `trend_windows`
+
+- Source: `document_impact_records` joined with `document_intelligence` and `documents`
+- Signals include: sentiment, impact score, catalyst type, confidence, novelty, source credibility
+- Weight computation factors: recency decay, source credibility, novelty score, extraction confidence
+- Always active (no toggle — this is the base layer)
+
+### Layer 2: Macro Signals
+
+**Data flow**: Macro news → Ingestion → Parsing → Event Classification (`global_events`) → Macro Interpolation (`macro_impact_records`) → `WeightedSignal` → `trend_windows`
+
+- Source: `macro_impact_records` joined with `global_events`
+- Each global event is classified by severity, affected regions/sectors/commodities, and estimated duration
+- Impact interpolation uses company exposure profiles (`exposure_profiles`) to compute per-company macro impact scores
+- Signals are weighted by `MACRO_SIGNAL_WEIGHT` (default: 0.3)
+
+**Toggle**: `macro_enabled` in `risk_configs` table (default: `true`)
+- When disabled: ingestion and classification continue (data preserved), but macro signals are excluded from aggregation
+- When re-enabled: resumes using the most recent events, including those classified while disabled
+
+**Weight configuration**:
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `MACRO_SIGNAL_WEIGHT` | `0.3` | Relative weight of macro vs company signals |
+| `MACRO_ENABLED` | `true` | Runtime toggle (also overridable via `risk_configs` DB table) |
+| `MACRO_CONFIDENCE_THRESHOLD` | `0.4` | Minimum confidence for event inclusion |
+| `MACRO_SHORT_TERM_STALENESS_HOURS` | `48` | Accelerated decay for short-term events |
+| `PROJECTION_CONFIDENCE_THRESHOLD` | `0.3` | Minimum confidence for projections |
+
+### Layer 3: Competitive Signals
+
+**Data flow**: Document intelligence → Pattern Matcher → Signal Propagation (`competitive_signal_records`) → `WeightedSignal` → `trend_windows`
+
+- Source: `competitive_signal_records` targeting a ticker
+- After aggregation completes for a company, the system propagates signals to competitors via `competitor_relationships`
+- Historical pattern matching (`find_self_patterns`, `find_cross_company_patterns`) informs propagation confidence
+- Signals are weighted by `COMPETITIVE_SIGNAL_WEIGHT` (default: 0.2)
+
+**Toggle**: `competitive_enabled` in `risk_configs` table (default: `true`)
+- When disabled: signal propagation is skipped; aggregation uses only company + macro signals
+- When re-enabled: resumes propagation using existing competitor relationships
+
+**Weight configuration**:
+
+| Environment Variable | Default | Description |
+|---|---|---|
+| `COMPETITIVE_SIGNAL_WEIGHT` | `0.2` | Relative weight of competitive signals |
+| `COMPETITIVE_ENABLED` | `true` | Runtime toggle (also overridable via `risk_configs` DB table) |
+| `COMPETITIVE_PATTERN_CONFIDENCE_THRESHOLD` | `0.3` | Minimum pattern confidence |
+| `COMPETITIVE_PROPAGATION_STRENGTH_THRESHOLD` | `0.2` | Minimum propagation strength |
+| `COMPETITIVE_ROUTINE_LOOKBACK_DAYS` | `180` | Lookback for routine signal patterns |
+| `COMPETITIVE_MAJOR_DECISION_LOOKBACK_DAYS` | `365` | Lookback for major corporate decisions |
+| `COMPETITIVE_MAJOR_DECISION_WEIGHT_MULTIPLIER` | `1.3` | Weight boost for major decisions |
+| `COMPETITIVE_STALENESS_WINDOW_DAYS` | `180` | Staleness window for pattern decay |
+| `COMPETITIVE_STALENESS_RECENT_DAYS` | `90` | Recent window (no decay) |
+| `COMPETITIVE_STALENESS_DECAY_PENALTY` | `0.5` | Decay penalty for stale patterns |
+| `COMPETITIVE_MIN_PATTERN_SAMPLES` | `3` | Minimum samples for pattern confidence |
+| `COMPETITIVE_PROPAGATION_FAILURE_THRESHOLD` | `5` | Consecutive failures before operator alert |
+
+### Safety Mechanisms
+
+- **Pattern-only suppression**: Trend shifts driven solely by competitive patterns are forced to `informational` mode
+- **Macro-only suppression**: Trend shifts driven solely by macro signals are forced to `informational` mode
+- **Consecutive failure alerting**: Both macro classification and competitive propagation track consecutive failures and emit `CRITICAL` alerts when thresholds are exceeded
+- **Graceful degradation**: If a signal layer fails, the system continues with the remaining layers
+
+---
+
+## Trading Engine Features
+
+### Position Sizing
+
+The trading engine uses a multi-factor position sizing algorithm:
+
+- **Risk-tier-based allocation**: Each risk tier (`conservative`, `moderate`, `aggressive`) defines maximum position size as a percentage of the active pool
+- **Gradual entry**: Large positions are split into configurable tranches (`TRADING_GRADUAL_ENTRY_TRANCHES`, default: 3) to reduce timing risk
+- **Absolute position cap**: Hard limit on any single position (`TRADING_ABSOLUTE_POSITION_CAP`, default: $50)
+- **Active pool minimum**: Orders are rejected if the active pool falls below `TRADING_ACTIVE_POOL_MINIMUM` (default: $100)
+- **Max open positions**: Configurable limit on concurrent positions (`TRADING_MAX_OPEN_POSITIONS`, default: 10)
+
+### Circuit Breakers
+
+Circuit breakers automatically halt trading when risk thresholds are breached:
+
+| Trigger Type | Description |
+|---|---|
+| `daily_loss` | Triggered when daily portfolio loss exceeds the emergency drawdown threshold (`TRADING_EMERGENCY_DRAWDOWN_THRESHOLD_PCT`, default: 40%) |
+| `single_position` | Triggered when a single position loss exceeds configured limits |
+| `volatility` | Triggered during extreme market volatility |
+| `manual` | Operator-triggered via the API |
+
+Circuit breaker events are persisted to `circuit_breaker_events` and state is tracked in Redis (`stonks:trading:circuit_breaker:*`).
+
+### Reserve Pool
+
+The reserve pool is a capital buffer that protects against drawdowns:
+
+- **Profit siphoning**: A configurable percentage of realized profits (`TRADING_RESERVE_SIPHON_PCT`, default: 20%) is automatically transferred to the reserve pool
+- **High-water mark**: When the reserve pool exceeds `TRADING_RESERVE_HIGH_WATER_PCT` (default: 30%) of total portfolio value, excess is returned to the active pool
+- **Emergency liquidation**: During severe drawdowns, the reserve pool can be used to cover losses
+- **Ledger tracking**: All reserve pool transactions are recorded in `reserve_pool_ledger` with trigger type and notes
+
+### Risk Tier Auto-Adjustment
+
+The engine periodically evaluates portfolio performance and adjusts the risk tier:
+
+- **Tiers**: `conservative` (lowest risk), `moderate` (default), `aggressive` (highest risk)
+- **Evaluation**: Based on recent win rate, drawdown, and portfolio heat
+- **History**: All tier changes are recorded in `risk_tier_history` for audit
+- **Configuration**: Tier can be set manually via `PUT /api/trading/config` or auto-adjusted by the engine
+
+**Risk Tier Defaults**:
+
+| Parameter | Conservative | Moderate | Aggressive |
+|---|---|---|---|
+| Min Confidence | 0.75 | 0.55 | 0.40 |
+| Max Position % | 5% | 10% | 15% |
+| Stop-Loss ATR Multiplier | 1.5 | 2.0 | 2.5 |
+| Reward/Risk Ratio | 2.0 | 1.5 | 1.2 |
+| Max Sector % | 20% | 30% | 40% |
+| Max Portfolio Heat | 10% | 20% | 30% |
+
+### Backtesting
+
+The trading engine supports historical backtesting:
+
+- **Launch**: `POST /api/trading/backtest` with start/end dates, initial capital, and risk tier
+- **Execution**: Runs asynchronously using `BacktestReplay` against historical recommendation data
+- **Results**: Stored in `backtest_runs` (summary metrics) and `backtest_trades` (individual trades)
+- **Metrics**: Total return, Sharpe ratio, max drawdown, win rate, profit factor, equity curve
+- **Polling**: `GET /api/trading/backtest/{id}` to check status and retrieve results
+
+### Notifications
+
+The engine supports two notification channels:
+
+| Channel | Configuration | Rate Limiting |
+|---|---|---|
+| SMS (AWS SNS) | `TRADING_SNS_TOPIC_ARN`, `TRADING_SNS_PHONE_NUMBER` | Redis-based per-channel rate limiting |
+| Email (Gmail) | `TRADING_GMAIL_SENDER`, `TRADING_GMAIL_RECIPIENT` | Redis-based per-channel rate limiting |
+
+Notification configuration can be updated at runtime via `PUT /api/trading/notifications/config`. History is available via `GET /api/trading/notifications/history`.
+
+### Micro-Trading
+
+An optional micro-trading mode for small, frequent trades:
+
+| Setting | Default | Description |
+|---|---|---|
+| `TRADING_MICRO_TRADING_ENABLED` | `false` | Enable micro-trading |
+| `TRADING_MICRO_TRADING_INTERVAL_SECONDS` | `300` | Polling interval |
+| `TRADING_MICRO_TRADING_ALLOCATION_CAP_PCT` | `0.03` | Max allocation per trade (3%) |
+| `TRADING_MICRO_TRADING_MAX_DAILY` | `10` | Max trades per day |
+| `TRADING_MICRO_TRADING_MAX_HOLD_MINUTES` | `120` | Max hold time |
+
+### Engine Control
+
+| Endpoint | Method | Description |
+|---|---|---|
+| `/api/trading/status` | GET | Current engine state |
+| `/api/trading/config` | PUT | Update configuration |
+| `/api/trading/pause` | POST | Pause the engine |
+| `/api/trading/resume` | POST | Resume the engine |
+| `/api/trading/reset` | POST | Full paper trading reset |
+| `/api/trading/debug` | GET | Diagnostic state dump |
+| `/api/trading/override/order` | POST | Submit manual override order |
+
+### Decision Loop
+
+The engine runs five concurrent async tasks:
+
+1. **Decision loop**: Polls recommendations, evaluates position sizing, submits orders
+2. **Stop-loss monitor**: Checks positions against stop-loss and take-profit levels at configurable intervals
+3. **Performance loop**: Computes daily snapshots, siphons profits to reserve pool
+4. **Risk tier scheduler**: Periodically evaluates and adjusts risk tier based on performance
+5. **Rebalance scheduler**: Evaluates portfolio rebalancing needs and computes position correlation matrix
+
+---
+
+## Shared Configuration Reference
+
+All services load configuration from environment variables via `services/shared/config.py`. The following variables are common across most services:
+
+### PostgreSQL
+
+| Variable | Default | Description |
+|---|---|---|
+| `POSTGRES_HOST` | `localhost` | Database host |
+| `POSTGRES_PORT` | `5432` | Database port |
+| `POSTGRES_DB` | `stonks` | Database name (auto-derived from `DEPLOY_STAGE` if empty) |
+| `POSTGRES_USER` | `stonks` | Database user |
+| `POSTGRES_PASSWORD` | `stonks_dev` | Database password |
+
+### Redis
+
+| Variable | Default | Description |
+|---|---|---|
+| `REDIS_HOST` | `localhost` | Redis host |
+| `REDIS_PORT` | `6379` | Redis port |
+| `REDIS_DB` | `0` | Redis database number |
+| `REDIS_PASSWORD` | _(none)_ | Redis password |
+
+### MinIO
+
+| Variable | Default | Description |
+|---|---|---|
+| `MINIO_ENDPOINT` | `localhost:9000` | MinIO endpoint |
+| `MINIO_ACCESS_KEY` | `minioadmin` | Access key |
+| `MINIO_SECRET_KEY` | `minioadmin` | Secret key |
+| `MINIO_SECURE` | `false` | Use HTTPS |
+
+### Ollama
+
+| Variable | Default | Description |
+|---|---|---|
+| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama API endpoint |
+| `OLLAMA_MODEL` | `qwen3.5:9b` | Default model |
+| `OLLAMA_TIMEOUT` | `120` | Request timeout (seconds) |
+| `OLLAMA_MAX_RETRIES` | `2` | Max retry attempts |
+
+### Observability
+
+| Variable | Default | Description |
+|---|---|---|
+| `LOG_LEVEL` | `INFO` | Logging level |
+| `JSON_LOGS` | `true` | Enable structured JSON logging |
+| `DEPLOY_STAGE` | _(empty)_ | Stage prefix for Redis keys and MinIO buckets |
diff --git a/pipelines/woodpecker/cleanup-step-secrets.yaml b/pipelines/woodpecker/cleanup-step-secrets.yaml
new file mode 100644
index 0000000..efe51f8
--- /dev/null
+++ b/pipelines/woodpecker/cleanup-step-secrets.yaml
@@ -0,0 +1,63 @@
+# CronJob + RBAC to clean up orphaned Woodpecker step secrets (wp-*-step-secret)
+# These accumulate when builds fail or are cancelled before cleanup runs.
+# Runs every 6 hours. TTL auto-deletes completed Job pods after 5 minutes.
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+ name: wp-secret-cleanup
+ namespace: woodpecker
+rules:
+ - apiGroups: [""]
+ resources: ["secrets"]
+ verbs: ["list", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+ name: wp-secret-cleanup
+ namespace: woodpecker
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: Role
+ name: wp-secret-cleanup
+subjects:
+ - kind: ServiceAccount
+ name: default
+ namespace: woodpecker
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+ name: cleanup-wp-step-secrets
+ namespace: woodpecker
+spec:
+ schedule: "0 */6 * * *"
+ successfulJobsHistoryLimit: 1
+ failedJobsHistoryLimit: 1
+ jobTemplate:
+ spec:
+ ttlSecondsAfterFinished: 300
+ template:
+ spec:
+ serviceAccountName: default
+ restartPolicy: Never
+ containers:
+ - name: cleanup
+ image: registry.celestium.life/dockerhub-cache/bitnami/kubectl:latest
+ command:
+ - /bin/sh
+ - -c
+ - |
+ echo 'Cleaning up orphaned Woodpecker step secrets...'
+ SECRETS=$(kubectl get secret -n woodpecker -o name | grep 'wp-.*step-secret')
+ COUNT=$(echo "$SECRETS" | grep -c 'step-secret' || true)
+ echo "Found $COUNT orphaned step secrets"
+ if [ "$COUNT" -gt 0 ]; then
+ echo "$SECRETS" | while read s; do
+ kubectl delete -n woodpecker "$s" 2>/dev/null || true
+ done
+ echo "Cleanup complete"
+ else
+ echo "Nothing to clean"
+ fi
diff --git a/tests/test_ingestion_unit.py b/tests/test_ingestion_unit.py
new file mode 100644
index 0000000..29751ef
--- /dev/null
+++ b/tests/test_ingestion_unit.py
@@ -0,0 +1,820 @@
+"""Unit tests for ingestion worker process_job function.
+
+Covers: successful job processing, adapter error with retry,
+retry exhaustion → dead-letter queue (via record_retrieval_failure),
+content hash deduplication skip, cross-source dedup via dedupe_items,
+and error handling paths (unexpected exceptions).
+
+Requirements: 2.1, 2.2, 2.3, 2.4
+"""
+from __future__ import annotations
+
+import uuid
+from datetime import datetime, timezone
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from services.adapters.base import AdapterResult
+from services.shared.redis_keys import (
+ QUEUE_PARSING,
+ queue_key,
+)
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _run_id() -> uuid.UUID:
+ return uuid.UUID("00000000-0000-0000-0000-000000000001")
+
+
+def _make_job(
+ source_type: str = "news_api",
+ ticker: str = "AAPL",
+ source_id: str = "src-1",
+ company_id: str = "cid-1",
+ config: dict | None = None,
+) -> dict:
+ return {
+ "source_type": source_type,
+ "ticker": ticker,
+ "source_id": source_id,
+ "company_id": company_id,
+ "config": config or {},
+ }
+
+
+def _make_adapter_result(
+ source_type: str = "news_api",
+ ticker: str = "AAPL",
+ items: list | None = None,
+ raw_payload: bytes = b'{"data": []}',
+ content_hash: str = "abc123",
+ error: str | None = None,
+) -> AdapterResult:
+ return AdapterResult(
+ source_type=source_type,
+ ticker=ticker,
+ items=items if items is not None else [{"title": "Test Article", "url": "https://example.com/1"}],
+ raw_payload=raw_payload,
+ content_hash=content_hash,
+ fetched_at=datetime(2026, 6, 15, 12, 0, 0, tzinfo=timezone.utc),
+ error=error,
+ )
+
+
+def _mock_pool() -> AsyncMock:
+ pool = AsyncMock()
+ pool.fetchval = AsyncMock(return_value=_run_id())
+ pool.execute = AsyncMock(return_value="UPDATE 1")
+ pool.fetchrow = AsyncMock(return_value=None)
+ pool.fetch = AsyncMock(return_value=[])
+ return pool
+
+
+def _mock_redis() -> AsyncMock:
+ rds = AsyncMock()
+ rds.rpush = AsyncMock(return_value=1)
+ rds.set = AsyncMock(return_value=True)
+ rds.get = AsyncMock(return_value=None)
+ rds.lpop = AsyncMock(return_value=None)
+ return rds
+
+
+def _mock_minio() -> MagicMock:
+ return MagicMock()
+
+
+def _mock_adapter(result: AdapterResult | None = None) -> AsyncMock:
+ adapter = AsyncMock()
+ adapter.fetch = AsyncMock(return_value=result or _make_adapter_result())
+ return adapter
+
+
+# ---------------------------------------------------------------------------
+# Test: Successful job processing
+# ---------------------------------------------------------------------------
+
+
+class TestSuccessfulJobProcessing:
+ """Verify the happy path: adapter returns items, they are persisted and enqueued."""
+
+ @pytest.mark.asyncio
+ async def test_successful_news_ingestion(self):
+ """A news_api job with new items should persist, enqueue for parsing, and mark complete."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ items = [
+ {"title": "Article 1", "url": "https://example.com/1"},
+ {"title": "Article 2", "url": "https://example.com/2"},
+ ]
+ result = _make_adapter_result(items=items, content_hash="hash1")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ new_doc_ids = ["doc-1", "doc-2"]
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path") as mock_upload,
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(2, new_doc_ids)) as mock_persist,
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(items, [])) as mock_dedupe,
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock) as mock_mark,
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock) as mock_reset,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Adapter was called
+ adapter.fetch.assert_awaited_once_with("AAPL", {})
+
+ # Ingestion run was created
+ pool.fetchval.assert_awaited_once()
+
+ # Raw artifact uploaded to MinIO
+ mock_upload.assert_called_once()
+
+ # Content hash was checked in Redis and then set
+ rds.get.assert_awaited()
+ rds.set.assert_awaited()
+
+ # Cross-source dedupe was called (news_api is not market_api/broker)
+ mock_dedupe.assert_awaited_once()
+
+ # Items were persisted
+ mock_persist.assert_awaited_once()
+
+ # New docs enqueued for parsing
+ assert rds.rpush.await_count == len(new_doc_ids)
+ for call_args in rds.rpush.await_args_list:
+ assert call_args[0][0] == queue_key(QUEUE_PARSING)
+
+ # mark_as_seen called for each new item
+ assert mock_mark.await_count == len(new_doc_ids)
+
+ # Retry state reset after success
+ mock_reset.assert_awaited_once_with(pool, "src-1")
+
+ # Ingestion run updated to completed
+ update_calls = [
+ c for c in pool.execute.await_args_list
+ if "completed" in str(c)
+ ]
+ assert len(update_calls) >= 1
+
+
+# ---------------------------------------------------------------------------
+# Test: Adapter error with retry
+# ---------------------------------------------------------------------------
+
+
+class TestAdapterErrorWithRetry:
+ """Verify that adapter errors are recorded as retrieval failures."""
+
+ @pytest.mark.asyncio
+ async def test_adapter_error_records_failure(self):
+ """When adapter returns an error, record_retrieval_failure is called."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ result = _make_adapter_result(error="API rate limited")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.record_retrieval_failure", new_callable=AsyncMock) as mock_record,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Failure was recorded with the error message
+ mock_record.assert_awaited_once()
+ call_kwargs = mock_record.await_args
+ assert call_kwargs[1]["error_message"] == "API rate limited"
+ assert call_kwargs[1]["run_id"] == str(_run_id())
+ assert call_kwargs[1]["source_id"] == "src-1"
+
+ @pytest.mark.asyncio
+ async def test_adapter_error_does_not_persist_items(self):
+ """When adapter returns an error, no items should be persisted or enqueued."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ result = _make_adapter_result(error="Connection timeout")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.record_retrieval_failure", new_callable=AsyncMock),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock) as mock_persist,
+ patch("services.ingestion.worker.upload_raw_artifact") as mock_upload,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # No items persisted, no upload
+ mock_persist.assert_not_awaited()
+ mock_upload.assert_not_called()
+
+ # No parsing jobs enqueued
+ rds.rpush.assert_not_awaited()
+
+
+# ---------------------------------------------------------------------------
+# Test: Retry exhaustion → dead-letter queue
+# ---------------------------------------------------------------------------
+
+
+class TestRetryExhaustion:
+ """Verify that record_retrieval_failure handles retry exhaustion.
+
+ The DLQ routing is handled inside record_retrieval_failure in the
+ metadata module. The worker calls record_retrieval_failure on both
+ adapter errors and unexpected exceptions. We verify the worker
+ correctly delegates to record_retrieval_failure in both cases.
+ """
+
+ @pytest.mark.asyncio
+ async def test_adapter_error_delegates_to_record_failure(self):
+ """Adapter error path calls record_retrieval_failure which manages retry state."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ result = _make_adapter_result(error="Server error 500")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.record_retrieval_failure", new_callable=AsyncMock, return_value={"exhausted": True, "retry_count": 5}) as mock_record,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ mock_record.assert_awaited_once_with(
+ pool,
+ run_id=str(_run_id()),
+ source_id="src-1",
+ error_message="Server error 500",
+ )
+
+ @pytest.mark.asyncio
+ async def test_exception_delegates_to_record_failure(self):
+ """Unexpected exception path also calls record_retrieval_failure."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ adapter = AsyncMock()
+ adapter.fetch = AsyncMock(side_effect=RuntimeError("Unexpected crash"))
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.record_retrieval_failure", new_callable=AsyncMock) as mock_record,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ mock_record.assert_awaited_once()
+ call_kwargs = mock_record.await_args
+ assert "Unexpected crash" in call_kwargs[1]["error_message"]
+
+
+# ---------------------------------------------------------------------------
+# Test: Content hash deduplication skip
+# ---------------------------------------------------------------------------
+
+
+class TestContentHashDedup:
+ """Verify that a previously-seen content hash causes the job to skip processing."""
+
+ @pytest.mark.asyncio
+ async def test_duplicate_content_hash_skips_processing(self):
+ """When Redis reports the content hash is already seen, skip persistence."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ # Simulate content hash already in Redis
+ rds.get = AsyncMock(return_value=b"1")
+ minio_client = _mock_minio()
+
+ result = _make_adapter_result(content_hash="already-seen-hash")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock) as mock_persist,
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock) as mock_dedupe,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Items should NOT be persisted
+ mock_persist.assert_not_awaited()
+
+ # Cross-source dedupe should NOT be called
+ mock_dedupe.assert_not_awaited()
+
+ # No parsing jobs enqueued
+ rds.rpush.assert_not_awaited()
+
+ # Ingestion run updated with items_new=0
+ update_calls = [
+ c for c in pool.execute.await_args_list
+ if "items_new=0" in str(c)
+ ]
+ assert len(update_calls) == 1
+
+ @pytest.mark.asyncio
+ async def test_no_content_hash_skips_dedupe_check(self):
+ """When content_hash is empty, the Redis dedupe check is skipped."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ items = [{"title": "Article"}]
+ result = _make_adapter_result(items=items, content_hash="")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(1, ["doc-1"])),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(items, [])),
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock),
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Redis get should not be called for dedupe (empty content_hash)
+ rds.get.assert_not_awaited()
+
+ # But items should still be persisted
+ # (persist_ingestion_items was called via the patch)
+
+
+# ---------------------------------------------------------------------------
+# Test: Cross-source dedup via dedupe_items
+# ---------------------------------------------------------------------------
+
+
+class TestCrossSourceDedup:
+ """Verify cross-source deduplication partitions items correctly."""
+
+ @pytest.mark.asyncio
+ async def test_dedupe_items_filters_duplicates(self):
+ """When dedupe_items finds duplicates, only new items are persisted."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ all_items = [
+ {"title": "New Article", "url": "https://example.com/new"},
+ {"title": "Dup Article", "url": "https://example.com/dup", "_dedupe_existing_id": "existing-doc"},
+ ]
+ new_items = [all_items[0]]
+ dup_items = [all_items[1]]
+
+ result = _make_adapter_result(items=all_items, content_hash="hash-x")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(new_items, dup_items)) as mock_dedupe,
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(1, ["doc-new"])) as mock_persist,
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock),
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # dedupe_items was called with all items
+ mock_dedupe.assert_awaited_once_with(pool, rds, all_items)
+
+ # persist_ingestion_items received only the new items
+ persist_call = mock_persist.await_args
+ assert persist_call[1]["items"] == new_items
+
+ @pytest.mark.asyncio
+ async def test_market_api_skips_cross_source_dedupe(self):
+ """Market API jobs should NOT go through cross-source deduplication."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ items = [{"close": 150.0, "volume": 1000000}]
+ result = _make_adapter_result(
+ source_type="market_api",
+ items=items,
+ content_hash="market-hash",
+ )
+ adapter = _mock_adapter(result)
+ adapters = {"market_api": adapter}
+ job = _make_job(source_type="market_api")
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock) as mock_dedupe,
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(1, ["snap-1"])),
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Cross-source dedupe should NOT be called for market_api
+ mock_dedupe.assert_not_awaited()
+
+ # No parsing jobs enqueued for market data
+ rds.rpush.assert_not_awaited()
+
+
+# ---------------------------------------------------------------------------
+# Test: Error handling paths
+# ---------------------------------------------------------------------------
+
+
+class TestErrorHandling:
+ """Verify error handling for unexpected exceptions and missing adapters."""
+
+ @pytest.mark.asyncio
+ async def test_unknown_source_type_returns_early(self):
+ """A job with no matching adapter should log a warning and return."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+ adapters = {"news_api": _mock_adapter()}
+ job = _make_job(source_type="unknown_type")
+
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # No ingestion run should be created for unknown adapter
+ pool.fetchval.assert_not_awaited()
+
+ @pytest.mark.asyncio
+ async def test_unexpected_exception_records_failure(self):
+ """An unexpected exception during processing records the failure."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ adapter = AsyncMock()
+ adapter.fetch = AsyncMock(side_effect=ConnectionError("DB gone"))
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.record_retrieval_failure", new_callable=AsyncMock) as mock_record,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ mock_record.assert_awaited_once()
+ call_kwargs = mock_record.await_args
+ assert "DB gone" in call_kwargs[1]["error_message"]
+ assert call_kwargs[1]["source_id"] == "src-1"
+
+ @pytest.mark.asyncio
+ async def test_upload_failure_records_error(self):
+ """If MinIO upload raises, the exception handler records the failure."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ result = _make_adapter_result()
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", side_effect=OSError("MinIO unreachable")),
+ patch("services.ingestion.worker.record_retrieval_failure", new_callable=AsyncMock) as mock_record,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ mock_record.assert_awaited_once()
+ assert "MinIO unreachable" in mock_record.await_args[1]["error_message"]
+
+
+# ---------------------------------------------------------------------------
+# Edge-case tests (Task 2.2)
+# Requirements: 2.1, 2.4
+# ---------------------------------------------------------------------------
+
+
+class TestEmptyAdapterResponse:
+ """Verify behaviour when the adapter returns zero items but no error."""
+
+ @pytest.mark.asyncio
+ async def test_empty_items_still_uploads_artifact(self):
+ """An adapter returning items=[] should upload the raw artifact and complete."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ result = _make_adapter_result(items=[], content_hash="empty-hash")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path") as mock_upload,
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(0, [])) as mock_persist,
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=([], [])) as mock_dedupe,
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Raw artifact should still be uploaded
+ mock_upload.assert_called_once()
+
+ # Cross-source dedupe called with empty list
+ mock_dedupe.assert_awaited_once_with(pool, rds, [])
+
+ # Persist called with empty items
+ mock_persist.assert_awaited_once()
+
+ # No parsing jobs enqueued (no new doc IDs)
+ rds.rpush.assert_not_awaited()
+
+ # Ingestion run marked completed
+ update_calls = [
+ c for c in pool.execute.await_args_list
+ if "completed" in str(c)
+ ]
+ assert len(update_calls) >= 1
+
+ @pytest.mark.asyncio
+ async def test_empty_items_with_no_content_hash(self):
+ """Empty items and empty content_hash should skip dedupe check and complete."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ result = _make_adapter_result(items=[], content_hash="")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(0, [])),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=([], [])),
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Redis get should NOT be called (empty content_hash skips dedupe)
+ rds.get.assert_not_awaited()
+ # Redis set should NOT be called (no hash to store)
+ rds.set.assert_not_awaited()
+
+
+class TestPartialPersistenceFailures:
+ """Verify behaviour when persist_ingestion_items returns fewer IDs than items."""
+
+ @pytest.mark.asyncio
+ async def test_partial_persist_enqueues_only_new_docs(self):
+ """If 3 items are passed but only 2 are new, only 2 parsing jobs are enqueued."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ items = [
+ {"title": "Article A", "url": "https://example.com/a"},
+ {"title": "Article B", "url": "https://example.com/b"},
+ {"title": "Article C", "url": "https://example.com/c"},
+ ]
+ result = _make_adapter_result(items=items, content_hash="partial-hash")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ # persist returns only 2 new IDs (one item was a DB-level duplicate)
+ new_ids = ["doc-a", "doc-c"]
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(items, [])),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(2, new_ids)),
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock) as mock_mark,
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Only 2 parsing jobs enqueued (matching new_ids count)
+ assert rds.rpush.await_count == 2
+
+ # mark_as_seen called for each new doc
+ assert mock_mark.await_count == 2
+
+
+class TestMultipleItemsSingleJob:
+ """Verify correct handling of multiple items in a single ingestion job."""
+
+ @pytest.mark.asyncio
+ async def test_multiple_items_all_enqueued_for_parsing(self):
+ """Five new items should produce five parsing queue entries."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ items = [{"title": f"Article {i}", "url": f"https://example.com/{i}"} for i in range(5)]
+ new_ids = [f"doc-{i}" for i in range(5)]
+ result = _make_adapter_result(items=items, content_hash="multi-hash")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(items, [])),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(5, new_ids)),
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock) as mock_mark,
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # All 5 items enqueued for parsing
+ assert rds.rpush.await_count == 5
+ for call_args in rds.rpush.await_args_list:
+ assert call_args[0][0] == queue_key(QUEUE_PARSING)
+
+ # mark_as_seen called for each
+ assert mock_mark.await_count == 5
+
+ # Ingestion run updated with correct counts
+ update_calls = [
+ c for c in pool.execute.await_args_list
+ if "completed" in str(c) and "items_new" in str(c)
+ ]
+ assert len(update_calls) == 1
+
+ @pytest.mark.asyncio
+ async def test_published_utc_tracking_picks_latest(self):
+ """For news_api jobs, the latest published_utc across items updates the source config."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ items = [
+ {"title": "Old", "url": "https://example.com/old", "published_utc": "2026-06-10T08:00:00Z"},
+ {"title": "New", "url": "https://example.com/new", "published_utc": "2026-06-15T12:00:00Z"},
+ {"title": "Mid", "url": "https://example.com/mid", "published_utc": "2026-06-12T10:00:00Z"},
+ ]
+ new_ids = ["doc-old", "doc-new", "doc-mid"]
+ result = _make_adapter_result(items=items, content_hash="pub-hash")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(items, [])),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(3, new_ids)),
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock),
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # The source config should be updated with the latest published_utc
+ source_update_calls = [
+ c for c in pool.execute.await_args_list
+ if "UPDATE sources" in str(c) and "last_published_at" in str(c)
+ ]
+ assert len(source_update_calls) == 1
+ # The JSON payload should contain the latest date
+ update_json = source_update_calls[0][0][1]
+ assert "2026-06-15T12:00:00Z" in update_json
+
+ @pytest.mark.asyncio
+ async def test_cross_source_dup_links_company_mention(self):
+ """When dedupe finds duplicates with existing IDs, the worker links them to the company."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ new_items = [{"title": "New Article", "url": "https://example.com/new"}]
+ dup_items = [
+ {"title": "Dup 1", "url": "https://example.com/dup1", "_dedupe_existing_id": "existing-doc-1"},
+ {"title": "Dup 2", "url": "https://example.com/dup2", "_dedupe_existing_id": "existing-doc-2"},
+ ]
+ all_items = new_items + dup_items
+
+ result = _make_adapter_result(items=all_items, content_hash="dup-link-hash")
+ adapter = _mock_adapter(result)
+ adapters = {"news_api": adapter}
+ job = _make_job()
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(new_items, dup_items)),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(1, ["doc-new"])),
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock),
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ patch("services.shared.metadata.persist_document_company_mention", new_callable=AsyncMock) as mock_mention,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # persist_document_company_mention called for each dup with existing ID
+ assert mock_mention.await_count == 2
+ linked_doc_ids = {call.kwargs["document_id"] for call in mock_mention.await_args_list}
+ assert linked_doc_ids == {"existing-doc-1", "existing-doc-2"}
+
+
+class TestMacroNewsEdgeCases:
+ """Verify edge cases specific to macro_news source type."""
+
+ @pytest.mark.asyncio
+ async def test_macro_news_without_company_id(self):
+ """Macro news jobs may lack company_id — the worker should handle this gracefully."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ items = [{"title": "Global Event", "url": "https://example.com/macro", "published_utc": "2026-06-15T12:00:00Z"}]
+ result = _make_adapter_result(
+ source_type="macro_news",
+ items=items,
+ content_hash="macro-hash",
+ )
+ adapter = _mock_adapter(result)
+ adapters = {"macro_news": adapter}
+ job = _make_job(source_type="macro_news")
+ # Remove company_id to simulate macro source
+ job.pop("company_id", None)
+
+ new_ids = ["doc-macro"]
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(items, [])),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(1, new_ids)),
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock),
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # Should complete without error — parsing job enqueued
+ assert rds.rpush.await_count == 1
+
+ # Ingestion run created with company_id=None
+ create_call = pool.fetchval.await_args
+ assert create_call[0][1] == "src-1" # source_id
+ assert create_call[0][2] is None # company_id is None
+
+ @pytest.mark.asyncio
+ async def test_macro_news_skips_dup_company_linking(self):
+ """Macro news should NOT link duplicate documents to companies (no company context)."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ minio_client = _mock_minio()
+
+ new_items = [{"title": "New Macro", "url": "https://example.com/new-macro"}]
+ dup_items = [{"title": "Dup Macro", "url": "https://example.com/dup-macro", "_dedupe_existing_id": "existing-macro"}]
+ all_items = new_items + dup_items
+
+ result = _make_adapter_result(
+ source_type="macro_news",
+ items=all_items,
+ content_hash="macro-dup-hash",
+ )
+ adapter = _mock_adapter(result)
+ adapters = {"macro_news": adapter}
+ job = _make_job(source_type="macro_news")
+
+ with (
+ patch("services.ingestion.worker.upload_raw_artifact", return_value="s3://bucket/path"),
+ patch("services.ingestion.worker.dedupe_items", new_callable=AsyncMock, return_value=(new_items, dup_items)),
+ patch("services.ingestion.worker.persist_ingestion_items", new_callable=AsyncMock, return_value=(1, ["doc-new-macro"])),
+ patch("services.ingestion.worker.mark_as_seen", new_callable=AsyncMock),
+ patch("services.ingestion.worker.reset_source_retry_state", new_callable=AsyncMock),
+ patch("services.shared.metadata.persist_document_company_mention", new_callable=AsyncMock) as mock_mention,
+ ):
+ from services.ingestion.worker import process_job
+ await process_job(job, pool, rds, minio_client, adapters)
+
+ # macro_news is excluded from dup company linking
+ mock_mention.assert_not_awaited()
diff --git a/tests/test_scheduler_unit.py b/tests/test_scheduler_unit.py
new file mode 100644
index 0000000..ee7dfbb
--- /dev/null
+++ b/tests/test_scheduler_unit.py
@@ -0,0 +1,861 @@
+"""Unit tests for scheduler pure functions and orchestration.
+
+Covers: get_cadence_for_source, compute_backoff, is_source_due,
+build_job_payload, schedule_cycle (mocked DB/Redis), check_rate_limit,
+recover_stale_documents, retry_failed_extractions, and error handling
+for DB/Redis connection failures.
+
+Requirements: 1.1, 1.2, 1.3, 1.4
+"""
+from __future__ import annotations
+
+import json
+import uuid
+from datetime import datetime, timedelta, timezone
+from unittest.mock import AsyncMock
+
+import pytest
+
+from services.scheduler.app import (
+ DEFAULT_BACKOFF_BASE,
+ DEFAULT_CADENCES,
+ DEFAULT_RATE_LIMITS,
+ MAX_BACKOFF,
+ MAX_RETRY_COUNT,
+ POLYGON_GLOBAL_RATE_LIMIT,
+ build_job_payload,
+ check_rate_limit,
+ compute_backoff,
+ get_cadence_for_source,
+ is_source_due,
+ recover_stale_documents,
+ retry_failed_extractions,
+ schedule_cycle,
+)
+from services.shared.redis_keys import (
+ QUEUE_EXTRACTION,
+ QUEUE_MACRO_CLASSIFICATION,
+ queue_key,
+)
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _now() -> datetime:
+ return datetime(2026, 6, 15, 12, 0, 0, tzinfo=timezone.utc)
+
+
+def _make_source(
+ source_id: str = "src-1",
+ company_id: str = "cid-1",
+ ticker: str = "AAPL",
+ source_type: str = "news_api",
+ source_name: str = "NewsAPI",
+ config: dict | None = None,
+ credibility_score: float = 0.8,
+ legal_name: str = "Apple Inc.",
+) -> dict:
+ """Build a dict that mimics an asyncpg.Record for a source row."""
+ return {
+ "source_id": source_id,
+ "company_id": company_id,
+ "ticker": ticker,
+ "legal_name": legal_name,
+ "source_type": source_type,
+ "source_name": source_name,
+ "config": config,
+ "credibility_score": credibility_score,
+ }
+
+
+def _make_last_run(
+ status: str = "completed",
+ started_at: datetime | None = None,
+ completed_at: datetime | None = None,
+ retry_count: int = 0,
+ next_retry_at: datetime | None = None,
+) -> dict:
+ """Build a dict that mimics an asyncpg.Record for an ingestion_runs row."""
+ return {
+ "status": status,
+ "started_at": started_at or _now() - timedelta(seconds=600),
+ "completed_at": completed_at or _now() - timedelta(seconds=600),
+ "retry_count": retry_count,
+ "next_retry_at": next_retry_at,
+ }
+
+
+def _mock_pool() -> AsyncMock:
+ """Create a mock asyncpg.Pool with standard async methods."""
+ pool = AsyncMock()
+ pool.fetch = AsyncMock(return_value=[])
+ pool.fetchrow = AsyncMock(return_value=None)
+ pool.fetchval = AsyncMock(return_value=None)
+ pool.execute = AsyncMock(return_value="UPDATE 0")
+ return pool
+
+
+def _mock_redis() -> AsyncMock:
+ """Create a mock redis.asyncio.Redis with standard async methods."""
+ rds = AsyncMock()
+ rds.rpush = AsyncMock(return_value=1)
+ rds.set = AsyncMock(return_value=True)
+ rds.get = AsyncMock(return_value=None)
+ rds.incr = AsyncMock(return_value=1)
+ rds.expire = AsyncMock(return_value=True)
+ rds.decr = AsyncMock(return_value=0)
+ rds.delete = AsyncMock(return_value=1)
+ return rds
+
+
+# ---------------------------------------------------------------------------
+# get_cadence_for_source
+# ---------------------------------------------------------------------------
+
+class TestGetCadenceForSource:
+ def test_returns_default_for_known_type(self):
+ assert get_cadence_for_source("market_api", None) == DEFAULT_CADENCES["market_api"]
+
+ def test_returns_fallback_for_unknown_type(self):
+ assert get_cadence_for_source("unknown_type", None) == 600
+
+ def test_config_override(self):
+ assert get_cadence_for_source("market_api", {"polling_interval_seconds": 120}) == 120
+
+ def test_config_override_clamped_to_minimum(self):
+ assert get_cadence_for_source("market_api", {"polling_interval_seconds": 3}) == 10
+
+ def test_invalid_config_value_falls_back(self):
+ assert get_cadence_for_source("news_api", {"polling_interval_seconds": "bad"}) == DEFAULT_CADENCES["news_api"]
+
+
+# ---------------------------------------------------------------------------
+# compute_backoff
+# ---------------------------------------------------------------------------
+
+class TestComputeBackoff:
+ def test_zero_retries(self):
+ assert compute_backoff(0) == DEFAULT_BACKOFF_BASE
+
+ def test_exponential_growth(self):
+ assert compute_backoff(1) == DEFAULT_BACKOFF_BASE * 2
+ assert compute_backoff(2) == DEFAULT_BACKOFF_BASE * 4
+
+ def test_capped_at_max(self):
+ assert compute_backoff(20) == MAX_BACKOFF
+
+ def test_exponent_capped_at_8(self):
+ # 2^8 = 256, so 60 * 256 = 15360 > MAX_BACKOFF (3600)
+ assert compute_backoff(8) == MAX_BACKOFF
+
+
+# ---------------------------------------------------------------------------
+# is_source_due
+# ---------------------------------------------------------------------------
+
+class TestIsSourceDue:
+ def test_never_run_is_due(self):
+ assert is_source_due("market_api", None, None, None, 0, None, _now())
+
+ def test_completed_within_cadence_not_due(self):
+ last = _now() - timedelta(seconds=100)
+ assert not is_source_due("market_api", None, last, "completed", 0, None, _now())
+
+ def test_completed_past_cadence_is_due(self):
+ last = _now() - timedelta(seconds=400)
+ assert is_source_due("market_api", None, last, "completed", 0, None, _now())
+
+ def test_running_not_due(self):
+ last = _now() - timedelta(seconds=5)
+ assert not is_source_due("market_api", None, last, "running", 0, None, _now())
+
+ def test_failed_within_backoff_not_due(self):
+ last = _now() - timedelta(seconds=30)
+ next_retry = _now() + timedelta(seconds=30)
+ assert not is_source_due("market_api", None, last, "failed", 1, next_retry, _now())
+
+ def test_failed_past_backoff_is_due(self):
+ last = _now() - timedelta(seconds=120)
+ next_retry = _now() - timedelta(seconds=10)
+ assert is_source_due("market_api", None, last, "failed", 1, next_retry, _now())
+
+ def test_failed_max_retries_not_due(self):
+ last = _now() - timedelta(seconds=120)
+ assert not is_source_due(
+ "market_api", None, last, "failed", MAX_RETRY_COUNT, None, _now()
+ )
+
+ def test_failed_no_next_retry_at_is_due(self):
+ """Failed with retries remaining and no next_retry_at → allow retry."""
+ last = _now() - timedelta(seconds=120)
+ assert is_source_due("market_api", None, last, "failed", 2, None, _now())
+
+
+# ---------------------------------------------------------------------------
+# build_job_payload
+# ---------------------------------------------------------------------------
+
+class TestBuildJobPayload:
+ def test_complete_payload(self):
+ src = _make_source()
+ now = _now()
+ job = build_job_payload(src, ["Apple", "AAPL Inc"], now)
+
+ assert job["source_id"] == "src-1"
+ assert job["company_id"] == "cid-1"
+ assert job["ticker"] == "AAPL"
+ assert job["legal_name"] == "Apple Inc."
+ assert job["aliases"] == ["Apple", "AAPL Inc"]
+ assert job["source_type"] == "news_api"
+ assert job["source_name"] == "NewsAPI"
+ assert job["config"] == {}
+ assert job["credibility_score"] == 0.8
+ assert job["scheduled_at"] == now.isoformat()
+
+ def test_null_company_id(self):
+ src = _make_source(company_id=None)
+ src["company_id"] = None
+ job = build_job_payload(src, [], _now())
+ assert job["company_id"] is None
+
+ def test_null_credibility_defaults_to_half(self):
+ src = _make_source(credibility_score=None)
+ src["credibility_score"] = None
+ job = build_job_payload(src, [], _now())
+ assert job["credibility_score"] == 0.5
+
+
+# ---------------------------------------------------------------------------
+# check_rate_limit (async)
+# ---------------------------------------------------------------------------
+
+class TestCheckRateLimit:
+ @pytest.mark.asyncio
+ async def test_allowed_when_under_limit(self):
+ rds = _mock_redis()
+ rds.incr = AsyncMock(return_value=1)
+ result = await check_rate_limit(rds, "news_api", _now())
+ assert result is True
+
+ @pytest.mark.asyncio
+ async def test_blocked_when_over_per_type_limit(self):
+ rds = _mock_redis()
+ limit = DEFAULT_RATE_LIMITS["news_api"]
+ rds.incr = AsyncMock(return_value=limit + 1)
+ result = await check_rate_limit(rds, "news_api", _now())
+ assert result is False
+
+ @pytest.mark.asyncio
+ async def test_polygon_global_limit_blocks(self):
+ """market_api is a Polygon type — global limit should block even if per-type is OK."""
+ rds = _mock_redis()
+ # Per-type counter is fine (1), but global counter exceeds limit
+ call_count = 0
+
+ async def _incr_side_effect(key):
+ nonlocal call_count
+ call_count += 1
+ if call_count == 1:
+ return 1 # per-type counter OK
+ return POLYGON_GLOBAL_RATE_LIMIT + 1 # global counter exceeded
+
+ rds.incr = AsyncMock(side_effect=_incr_side_effect)
+ result = await check_rate_limit(rds, "market_api", _now())
+ assert result is False
+ # Should have decremented the per-type counter
+ rds.decr.assert_called_once()
+
+ @pytest.mark.asyncio
+ async def test_non_polygon_type_skips_global_check(self):
+ """filings_api is not a Polygon type — no global limit check."""
+ rds = _mock_redis()
+ rds.incr = AsyncMock(return_value=1)
+ result = await check_rate_limit(rds, "filings_api", _now())
+ assert result is True
+ # incr should be called only once (per-type), not twice (no global)
+ assert rds.incr.call_count == 1
+
+ @pytest.mark.asyncio
+ async def test_expire_set_on_first_increment(self):
+ rds = _mock_redis()
+ rds.incr = AsyncMock(return_value=1)
+ await check_rate_limit(rds, "news_api", _now())
+ rds.expire.assert_called()
+
+ @pytest.mark.asyncio
+ async def test_custom_max_per_minute(self):
+ rds = _mock_redis()
+ rds.incr = AsyncMock(return_value=6)
+ result = await check_rate_limit(rds, "news_api", _now(), max_per_minute=5)
+ assert result is False
+
+
+# ---------------------------------------------------------------------------
+# schedule_cycle (mocked DB/Redis)
+# ---------------------------------------------------------------------------
+
+class TestScheduleCycle:
+ @pytest.mark.asyncio
+ async def test_enqueues_due_sources(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ src = _make_source()
+ pool.fetch = AsyncMock(side_effect=[
+ [src], # fetch_active_sources
+ [], # fetch_macro_sources
+ [], # fetch_global_market_sources
+ [], # fetch_aliases_for_company returns rows
+ ])
+ # fetch_last_run returns None (never run → due)
+ pool.fetchrow = AsyncMock(return_value=None)
+ # Rate limit OK
+ rds.incr = AsyncMock(return_value=1)
+
+ enqueued = await schedule_cycle(pool, rds)
+ assert enqueued == 1
+ rds.rpush.assert_called_once()
+
+ # Verify the enqueued payload
+ call_args = rds.rpush.call_args
+ payload = json.loads(call_args[0][1])
+ assert payload["source_id"] == "src-1"
+ assert payload["ticker"] == "AAPL"
+
+ @pytest.mark.asyncio
+ async def test_skips_not_due_sources(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ src = _make_source()
+ pool.fetch = AsyncMock(side_effect=[
+ [src], # fetch_active_sources
+ [], # fetch_macro_sources
+ [], # fetch_global_market_sources
+ ])
+ # Last run was recent → not due
+ pool.fetchrow = AsyncMock(return_value=_make_last_run(
+ status="completed",
+ completed_at=datetime.now(tz=timezone.utc) - timedelta(seconds=10),
+ ))
+
+ enqueued = await schedule_cycle(pool, rds)
+ assert enqueued == 0
+ rds.rpush.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_skips_rate_limited_sources(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ src = _make_source()
+ pool.fetch = AsyncMock(side_effect=[
+ [src], # fetch_active_sources
+ [], # fetch_macro_sources
+ [], # fetch_global_market_sources
+ ])
+ pool.fetchrow = AsyncMock(return_value=None) # never run → due
+ # Rate limit exceeded
+ rds.incr = AsyncMock(return_value=DEFAULT_RATE_LIMITS["news_api"] + 1)
+
+ enqueued = await schedule_cycle(pool, rds)
+ assert enqueued == 0
+
+ @pytest.mark.asyncio
+ async def test_enqueues_macro_sources(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ macro_src = _make_source(
+ source_id="macro-1",
+ company_id=None,
+ ticker="",
+ source_type="macro_news",
+ source_name="MacroNewsSource",
+ )
+ macro_src["company_id"] = None
+
+ pool.fetch = AsyncMock(side_effect=[
+ [], # fetch_active_sources (empty)
+ [macro_src], # fetch_macro_sources
+ [], # fetch_global_market_sources
+ ])
+ pool.fetchrow = AsyncMock(return_value=None) # never run → due
+ rds.incr = AsyncMock(return_value=1)
+
+ enqueued = await schedule_cycle(pool, rds)
+ assert enqueued == 1
+
+
+# ---------------------------------------------------------------------------
+# recover_stale_documents (mocked DB/Redis)
+# ---------------------------------------------------------------------------
+
+class TestRecoverStaleDocuments:
+ @pytest.mark.asyncio
+ async def test_recovers_stale_parsed_docs(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ doc_id = uuid.uuid4()
+ pool.fetch = AsyncMock(return_value=[
+ {"id": doc_id, "document_type": "news", "ticker": "AAPL"},
+ ])
+ # _enqueue_if_new: rds.set returns True (new marker)
+ rds.set = AsyncMock(return_value=True)
+
+ count = await recover_stale_documents(pool, rds)
+ assert count == 1
+ # Should push to extraction queue
+ rds.rpush.assert_called_once()
+ call_args = rds.rpush.call_args
+ assert queue_key(QUEUE_EXTRACTION) in call_args[0][0]
+ # Should update documents
+ pool.execute.assert_called_once()
+
+ @pytest.mark.asyncio
+ async def test_routes_macro_event_to_classification_queue(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ doc_id = uuid.uuid4()
+ pool.fetch = AsyncMock(return_value=[
+ {"id": doc_id, "document_type": "macro_event", "ticker": ""},
+ ])
+ rds.set = AsyncMock(return_value=True)
+
+ count = await recover_stale_documents(pool, rds)
+ assert count == 1
+ call_args = rds.rpush.call_args
+ assert queue_key(QUEUE_MACRO_CLASSIFICATION) in call_args[0][0]
+
+ @pytest.mark.asyncio
+ async def test_skips_already_enqueued_docs(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ doc_id = uuid.uuid4()
+ pool.fetch = AsyncMock(return_value=[
+ {"id": doc_id, "document_type": "news", "ticker": "AAPL"},
+ ])
+ # _enqueue_if_new: rds.set returns False (already tracked)
+ rds.set = AsyncMock(return_value=False)
+
+ count = await recover_stale_documents(pool, rds)
+ assert count == 0
+ rds.rpush.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_no_stale_docs_returns_zero(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+ pool.fetch = AsyncMock(return_value=[])
+
+ count = await recover_stale_documents(pool, rds)
+ assert count == 0
+
+
+# ---------------------------------------------------------------------------
+# retry_failed_extractions (mocked DB/Redis)
+# ---------------------------------------------------------------------------
+
+class TestRetryFailedExtractions:
+ @pytest.mark.asyncio
+ async def test_retries_failed_docs(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ doc_id = uuid.uuid4()
+ pool.fetch = AsyncMock(return_value=[
+ {"id": doc_id, "document_type": "filing", "ticker": "MSFT"},
+ ])
+ rds.set = AsyncMock(return_value=True)
+
+ count = await retry_failed_extractions(pool, rds)
+ assert count == 1
+ # Should push to extraction queue
+ rds.rpush.assert_called_once()
+ # Should delete failed intelligence rows and reset status
+ assert pool.execute.call_count == 2
+
+ @pytest.mark.asyncio
+ async def test_routes_macro_event_to_classification(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ doc_id = uuid.uuid4()
+ pool.fetch = AsyncMock(return_value=[
+ {"id": doc_id, "document_type": "macro_event", "ticker": ""},
+ ])
+ rds.set = AsyncMock(return_value=True)
+
+ count = await retry_failed_extractions(pool, rds)
+ assert count == 1
+ call_args = rds.rpush.call_args
+ assert queue_key(QUEUE_MACRO_CLASSIFICATION) in call_args[0][0]
+
+ @pytest.mark.asyncio
+ async def test_no_failed_docs_returns_zero(self):
+ pool = _mock_pool()
+ rds = _mock_redis()
+ pool.fetch = AsyncMock(return_value=[])
+
+ count = await retry_failed_extractions(pool, rds)
+ assert count == 0
+
+
+# ---------------------------------------------------------------------------
+# Error handling: DB/Redis connection failures
+# ---------------------------------------------------------------------------
+
+class TestErrorHandling:
+ @pytest.mark.asyncio
+ async def test_schedule_cycle_handles_db_failure(self):
+ """DB failure in fetch_active_sources should propagate but not crash the process."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ pool.fetch = AsyncMock(side_effect=Exception("connection refused"))
+
+ with pytest.raises(Exception, match="connection refused"):
+ await schedule_cycle(pool, rds)
+
+ @pytest.mark.asyncio
+ async def test_recover_stale_handles_db_failure(self):
+ """DB failure in recover_stale_documents should propagate."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+ pool.fetch = AsyncMock(side_effect=ConnectionError("pg pool exhausted"))
+
+ with pytest.raises(ConnectionError, match="pg pool exhausted"):
+ await recover_stale_documents(pool, rds)
+
+ @pytest.mark.asyncio
+ async def test_check_rate_limit_handles_redis_failure(self):
+ """Redis failure in check_rate_limit should propagate."""
+ rds = _mock_redis()
+ rds.incr = AsyncMock(side_effect=ConnectionError("redis unavailable"))
+
+ with pytest.raises(ConnectionError, match="redis unavailable"):
+ await check_rate_limit(rds, "news_api", _now())
+
+ @pytest.mark.asyncio
+ async def test_retry_failed_handles_redis_failure(self):
+ """Redis failure during enqueue should propagate."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ doc_id = uuid.uuid4()
+ pool.fetch = AsyncMock(return_value=[
+ {"id": doc_id, "document_type": "news", "ticker": "AAPL"},
+ ])
+ rds.set = AsyncMock(return_value=True)
+ rds.rpush = AsyncMock(side_effect=ConnectionError("redis down"))
+
+ with pytest.raises(ConnectionError, match="redis down"):
+ await retry_failed_extractions(pool, rds)
+
+
+# ===========================================================================
+# Edge-case unit tests — boundary conditions and rate limiting
+# Requirements: 1.3, 1.4
+# ===========================================================================
+
+
+# ---------------------------------------------------------------------------
+# get_cadence_for_source — boundary conditions
+# ---------------------------------------------------------------------------
+
+class TestGetCadenceEdgeCases:
+ def test_zero_polling_interval_clamped_to_minimum(self):
+ """Config with polling_interval_seconds=0 should be clamped to 10."""
+ assert get_cadence_for_source("news_api", {"polling_interval_seconds": 0}) == 10
+
+ def test_negative_polling_interval_clamped_to_minimum(self):
+ """Negative interval should be clamped to 10."""
+ assert get_cadence_for_source("market_api", {"polling_interval_seconds": -50}) == 10
+
+ def test_exactly_minimum_polling_interval(self):
+ """Interval of exactly 10 should be accepted as-is."""
+ assert get_cadence_for_source("market_api", {"polling_interval_seconds": 10}) == 10
+
+ def test_none_config_value_uses_default(self):
+ """Config with polling_interval_seconds=None should fall back to default."""
+ assert get_cadence_for_source("news_api", {"polling_interval_seconds": None}) == DEFAULT_CADENCES["news_api"]
+
+ def test_empty_config_dict_uses_default(self):
+ """Empty config dict (no polling_interval_seconds key) uses default."""
+ assert get_cadence_for_source("filings_api", {}) == DEFAULT_CADENCES["filings_api"]
+
+ def test_float_polling_interval_truncated(self):
+ """Float value should be truncated to int via int()."""
+ assert get_cadence_for_source("news_api", {"polling_interval_seconds": 120.9}) == 120
+
+
+# ---------------------------------------------------------------------------
+# compute_backoff — boundary conditions
+# ---------------------------------------------------------------------------
+
+class TestComputeBackoffEdgeCases:
+ def test_negative_retry_count(self):
+ """Negative retry count should still produce a valid backoff (2^negative → fraction, but int floors)."""
+ result = compute_backoff(-1)
+ # 2^min(-1, 8) = 2^-1 = 0.5, so 60 * 0.5 = 30
+ assert result == 30
+
+ def test_exactly_at_cap_boundary(self):
+ """Find the exact retry count where backoff first hits MAX_BACKOFF."""
+ # DEFAULT_BACKOFF_BASE=60, MAX_BACKOFF=3600
+ # 60 * 2^6 = 3840 > 3600, so retry_count=6 should hit the cap
+ assert compute_backoff(6) == MAX_BACKOFF
+
+ def test_just_below_cap(self):
+ """retry_count=5: 60 * 2^5 = 1920, below MAX_BACKOFF."""
+ assert compute_backoff(5) == DEFAULT_BACKOFF_BASE * 32 # 1920
+
+ def test_very_large_retry_count(self):
+ """Very large retry count should still be capped at MAX_BACKOFF."""
+ assert compute_backoff(1000) == MAX_BACKOFF
+
+
+# ---------------------------------------------------------------------------
+# is_source_due — boundary conditions
+# ---------------------------------------------------------------------------
+
+class TestIsSourceDueEdgeCases:
+ def test_exactly_at_max_retry_count_not_due(self):
+ """Exactly at MAX_RETRY_COUNT should NOT be due."""
+ last = _now() - timedelta(seconds=9999)
+ assert not is_source_due(
+ "market_api", None, last, "failed", MAX_RETRY_COUNT, None, _now()
+ )
+
+ def test_one_below_max_retry_count_is_due(self):
+ """One below MAX_RETRY_COUNT with no next_retry_at should be due."""
+ last = _now() - timedelta(seconds=9999)
+ assert is_source_due(
+ "market_api", None, last, "failed", MAX_RETRY_COUNT - 1, None, _now()
+ )
+
+ def test_completed_exactly_at_cadence_boundary(self):
+ """Completed exactly at cadence seconds ago should be due (elapsed >= cadence)."""
+ cadence = DEFAULT_CADENCES["market_api"] # 300
+ last = _now() - timedelta(seconds=cadence)
+ assert is_source_due("market_api", None, last, "completed", 0, None, _now())
+
+ def test_completed_one_second_before_cadence_not_due(self):
+ """Completed one second less than cadence ago should NOT be due."""
+ cadence = DEFAULT_CADENCES["market_api"] # 300
+ last = _now() - timedelta(seconds=cadence - 1)
+ assert not is_source_due("market_api", None, last, "completed", 0, None, _now())
+
+ def test_status_completed_but_completed_at_none_is_due(self):
+ """Status is not None but completed_at is None → should be due (falls through to cadence check with None)."""
+ assert is_source_due("market_api", None, None, "completed", 0, None, _now())
+
+ def test_next_retry_at_exactly_now_not_due(self):
+ """next_retry_at exactly equal to now → now < nra is False, so should be due."""
+ last = _now() - timedelta(seconds=120)
+ assert is_source_due("market_api", None, last, "failed", 1, _now(), _now())
+
+
+# ---------------------------------------------------------------------------
+# build_job_payload — edge cases
+# ---------------------------------------------------------------------------
+
+class TestBuildJobPayloadEdgeCases:
+ def test_config_as_json_string(self):
+ """Config stored as a JSON string should be coerced to a dict."""
+ src = _make_source(config='{"polling_interval_seconds": 120}')
+ src["config"] = '{"polling_interval_seconds": 120}'
+ job = build_job_payload(src, [], _now())
+ assert job["config"] == {"polling_interval_seconds": 120}
+
+ def test_config_as_invalid_json_string(self):
+ """Invalid JSON string config should fall back to empty dict."""
+ src = _make_source()
+ src["config"] = "not-json"
+ job = build_job_payload(src, [], _now())
+ assert job["config"] == {}
+
+ def test_empty_ticker(self):
+ """Source with empty ticker should produce empty string in payload."""
+ src = _make_source(ticker="")
+ job = build_job_payload(src, [], _now())
+ assert job["ticker"] == ""
+
+ def test_zero_credibility_score(self):
+ """Credibility score of 0.0 should be preserved (not treated as falsy → 0.5)."""
+ src = _make_source(credibility_score=0.0)
+ # 0.0 is falsy in Python, so this tests the boundary
+ job = build_job_payload(src, [], _now())
+ # The implementation uses `if source["credibility_score"]` which treats 0.0 as falsy
+ # This documents the actual behavior: 0.0 → 0.5
+ assert job["credibility_score"] == 0.5
+
+ def test_many_aliases(self):
+ """Multiple aliases should all be included in the payload."""
+ src = _make_source()
+ aliases = ["Apple Inc.", "Apple Computer", "AAPL", "Apple"]
+ job = build_job_payload(src, aliases, _now())
+ assert job["aliases"] == aliases
+
+
+# ---------------------------------------------------------------------------
+# check_rate_limit — boundary and edge cases
+# ---------------------------------------------------------------------------
+
+class TestCheckRateLimitEdgeCases:
+ @pytest.mark.asyncio
+ async def test_exactly_at_per_type_limit_allowed(self):
+ """Count exactly equal to the limit should be allowed (only > limit blocks)."""
+ rds = _mock_redis()
+ limit = DEFAULT_RATE_LIMITS["news_api"]
+ rds.incr = AsyncMock(return_value=limit)
+ result = await check_rate_limit(rds, "news_api", _now())
+ assert result is True
+
+ @pytest.mark.asyncio
+ async def test_one_over_per_type_limit_blocked(self):
+ """Count one over the limit should be blocked."""
+ rds = _mock_redis()
+ limit = DEFAULT_RATE_LIMITS["news_api"]
+ rds.incr = AsyncMock(return_value=limit + 1)
+ result = await check_rate_limit(rds, "news_api", _now())
+ assert result is False
+
+ @pytest.mark.asyncio
+ async def test_polygon_exactly_at_global_limit_allowed(self):
+ """Polygon global count exactly at limit should be allowed."""
+ rds = _mock_redis()
+ call_count = 0
+
+ async def _incr(key):
+ nonlocal call_count
+ call_count += 1
+ if call_count == 1:
+ return 1 # per-type OK
+ return POLYGON_GLOBAL_RATE_LIMIT # exactly at global limit
+
+ rds.incr = AsyncMock(side_effect=_incr)
+ result = await check_rate_limit(rds, "market_api", _now())
+ assert result is True
+ rds.decr.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_polygon_one_over_global_limit_blocked_and_decrements(self):
+ """Polygon global count one over limit should block and decrement per-type counter."""
+ rds = _mock_redis()
+ call_count = 0
+
+ async def _incr(key):
+ nonlocal call_count
+ call_count += 1
+ if call_count == 1:
+ return 1 # per-type OK
+ return POLYGON_GLOBAL_RATE_LIMIT + 1 # one over global limit
+
+ rds.incr = AsyncMock(side_effect=_incr)
+ result = await check_rate_limit(rds, "market_api", _now())
+ assert result is False
+ rds.decr.assert_called_once()
+
+ @pytest.mark.asyncio
+ async def test_unknown_source_type_uses_default_limit(self):
+ """Unknown source type should use the fallback limit of 30."""
+ rds = _mock_redis()
+ rds.incr = AsyncMock(return_value=30) # exactly at default limit
+ result = await check_rate_limit(rds, "unknown_type", _now())
+ assert result is True
+
+ @pytest.mark.asyncio
+ async def test_unknown_source_type_over_default_blocked(self):
+ """Unknown source type at 31 should be blocked (default limit is 30)."""
+ rds = _mock_redis()
+ rds.incr = AsyncMock(return_value=31)
+ result = await check_rate_limit(rds, "unknown_type", _now())
+ assert result is False
+
+ @pytest.mark.asyncio
+ async def test_expire_not_called_on_subsequent_increments(self):
+ """Expire should only be called when incr returns 1 (first increment)."""
+ rds = _mock_redis()
+ rds.incr = AsyncMock(return_value=5) # not the first increment
+ await check_rate_limit(rds, "filings_api", _now())
+ rds.expire.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_news_api_polygon_global_check(self):
+ """news_api is a Polygon type — should also check global limit."""
+ rds = _mock_redis()
+ call_count = 0
+
+ async def _incr(key):
+ nonlocal call_count
+ call_count += 1
+ return 1 # both counters OK
+
+ rds.incr = AsyncMock(side_effect=_incr)
+ result = await check_rate_limit(rds, "news_api", _now())
+ assert result is True
+ # Should have called incr twice: per-type + global
+ assert call_count == 2
+
+
+# ---------------------------------------------------------------------------
+# schedule_cycle — empty source list edge case
+# ---------------------------------------------------------------------------
+
+class TestScheduleCycleEdgeCases:
+ @pytest.mark.asyncio
+ async def test_empty_source_lists_returns_zero(self):
+ """When all source queries return empty lists, enqueued count should be 0."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ pool.fetch = AsyncMock(side_effect=[
+ [], # fetch_active_sources
+ [], # fetch_macro_sources
+ [], # fetch_global_market_sources
+ ])
+
+ enqueued = await schedule_cycle(pool, rds)
+ assert enqueued == 0
+ rds.rpush.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_multiple_sources_mixed_due_and_not_due(self):
+ """Mix of due and not-due sources: only due ones get enqueued."""
+ pool = _mock_pool()
+ rds = _mock_redis()
+
+ src1 = _make_source(source_id="src-1", ticker="AAPL")
+ src2 = _make_source(source_id="src-2", ticker="MSFT")
+
+ pool.fetch = AsyncMock(side_effect=[
+ [src1, src2], # fetch_active_sources
+ [], # fetch_macro_sources
+ [], # fetch_global_market_sources
+ [], # aliases for src-1
+ ])
+
+ # src-1 never run (due), src-2 recently completed (not due)
+ recent_run = _make_last_run(
+ status="completed",
+ completed_at=datetime.now(tz=timezone.utc) - timedelta(seconds=10),
+ )
+
+ call_count = 0
+
+ async def _fetchrow(*args, **kwargs):
+ nonlocal call_count
+ call_count += 1
+ if call_count == 1:
+ return None # src-1: never run → due
+ return recent_run # src-2: recently completed → not due
+
+ pool.fetchrow = AsyncMock(side_effect=_fetchrow)
+ rds.incr = AsyncMock(return_value=1)
+
+ enqueued = await schedule_cycle(pool, rds)
+ assert enqueued == 1