stonks-oracle/docs/architecture-kubernetes.md

# Kubernetes Architecture — Stonks Oracle

This document describes the Kubernetes deployment topology for Stonks Oracle, derived from the Helm chart at `infra/helm/stonks-oracle/`.

All application workloads deploy to the `stonks-oracle` namespace. External cluster services (PostgreSQL, Redis, MinIO, Ollama) run in their own namespaces and are referenced via cross-namespace DNS.

## Deployment Diagram

```mermaid
graph TB
    %% ── External traffic ──────────────────────────────────────────
    internet((Internet))

    subgraph traefik ["kube-system · Traefik Ingress Controller"]
        direction LR
        ing_dash["stonks.celestium.life"]
        ing_api["stonks-api.celestium.life"]
        ing_reg["stonks-registry.celestium.life"]
        ing_trade["stonks-trading.celestium.life"]
        ing_superset["stonks-dash.celestium.life"]
        ing_trino["stonks-trino.celestium.life"]
    end

    internet --> traefik

    %% ── stonks-oracle namespace ───────────────────────────────────
    subgraph ns ["stonks-oracle namespace"]
        direction TB

        %% ── API Tier (ingress-facing) ─────────────────────────────
        subgraph api_tier ["API Tier · tier: api"]
            direction LR
            query_api["query-api<br/><i>Deployment · 1 replica</i><br/>:8000<br/><i>readiness: /docs</i>"]
            symbol_registry["symbol-registry<br/><i>Deployment · 1 replica</i><br/>:8000<br/><i>readiness: /docs · liveness: /docs</i>"]
        end

        %% ── Frontend Tier ─────────────────────────────────────────
        subgraph frontend_tier ["Frontend Tier · tier: frontend"]
            dashboard["dashboard<br/><i>Deployment · 1 replica</i><br/>:8080<br/><i>nginx-unprivileged</i><br/><i>readiness: / · liveness: /</i>"]
        end

        %% ── Trading Tier ──────────────────────────────────────────
        subgraph trading_tier ["Trading Tier · tier: trading"]
            direction LR
            trading_engine["trading-engine<br/><i>Deployment · 1 replica</i><br/>:8000<br/><i>readiness: /ready · liveness: /health</i>"]
            risk_engine["risk-engine<br/><i>Deployment · 1 replica</i><br/>:8000"]
            broker_adapter["broker-adapter<br/><i>Deployment · 1 replica</i><br/><i>queue-driven worker · pipeline-gated</i>"]
        end

        %% ── Orchestration Tier ────────────────────────────────────
        subgraph orchestration_tier ["Orchestration Tier · tier: orchestration"]
            scheduler["scheduler<br/><i>Deployment · 1 replica · pipeline-gated</i><br/><i>init: migrations → seed → backfill</i>"]
        end

        %% ── Ingestion Tier ────────────────────────────────────────
        subgraph ingestion_tier ["Ingestion Tier · tier: ingestion"]
            ingestion["ingestion<br/><i>Deployment · 1 replica · pipeline-gated</i><br/><i>queue-driven worker</i>"]
        end

        %% ── Processing Tier (pipeline workers) ────────────────────
        subgraph processing_tier ["Processing Tier · tier: processing"]
            direction LR
            parser["parser<br/><i>Deployment · 2 replicas · pipeline-gated</i>"]
            extractor["extractor<br/><i>Deployment · 1 replica · pipeline-gated</i>"]
            aggregation["aggregation<br/><i>Deployment · 4 replicas · pipeline-gated</i>"]
            recommendation["recommendation<br/><i>Deployment · 1 replica · pipeline-gated</i>"]
        end

        %% ── Analytics Tier ────────────────────────────────────────
        subgraph analytics_tier ["Analytics Tier · tier: analytics"]
            direction LR
            lake_publisher["lake-publisher<br/><i>Deployment · 1 replica · pipeline-gated</i><br/><i>queue-driven worker</i>"]
            hive_metastore["hive-metastore<br/><i>Deployment · 1 replica</i><br/>:9083<br/><i>apache/hive:4.0.0</i><br/><i>PVC: hive-metastore-data</i>"]
            trino["trino<br/><i>Deployment · 1 replica</i><br/>:8080<br/><i>trinodb/trino:latest</i><br/><i>readiness: /v1/info</i>"]
        end

        %% ── Superset (tier: dashboard in template) ────────────────
        subgraph superset_block ["Superset · tier: dashboard"]
            superset["superset<br/><i>Deployment · 1 replica</i><br/>:8088<br/><i>custom image</i><br/><i>PVC: superset-data</i><br/><i>readiness: /health</i>"]
        end

        %% ── Helm Secrets ──────────────────────────────────────────
        subgraph secrets_block ["Helm-Managed Secrets"]
            direction LR
            sec_core["stonks-core-secrets<br/><i>POSTGRES_PASSWORD</i><br/><i>MINIO_ACCESS_KEY</i><br/><i>MINIO_SECRET_KEY</i><br/><i>REDIS_PASSWORD</i>"]
            sec_broker["stonks-broker-secrets<br/><i>BROKER_API_KEY</i><br/><i>BROKER_API_SECRET</i><br/><i>BROKER_BASE_URL</i>"]
            sec_market["stonks-market-secrets<br/><i>MARKET_DATA_API_KEY</i>"]
            sec_gmail["stonks-gmail-secrets<br/><i>GMAIL_SENDER</i><br/><i>GMAIL_RECIPIENT</i><br/><i>GMAIL_APP_PASSWORD</i>"]
            sec_dashboard["stonks-dashboard-secrets<br/><i>SUPERSET_SECRET_KEY</i><br/><i>SUPERSET_ADMIN_PASSWORD</i>"]
        end

        %% ── ConfigMap ─────────────────────────────────────────────
        configmap["stonks-config<br/><i>ConfigMap</i><br/><i>All env vars from values.yaml config block</i>"]
    end

    %% ── External Cluster Services ─────────────────────────────────
    subgraph pg_ns ["postgresql-service namespace"]
        postgres[("PostgreSQL<br/>postgresql-rw:5432")]
    end

    subgraph redis_ns ["redis-service namespace"]
        redis[("Redis<br/>redis-master:6379")]
    end

    subgraph minio_ns ["minio-service namespace"]
        minio[("MinIO<br/>minio:80")]
    end

    subgraph ollama_ns ["ollama-service namespace"]
        ollama[("Ollama<br/>ollama:11434<br/><i>GPU: 4070 Ti Super 16GB</i>")]
    end

    %% ── Ingress Routes ────────────────────────────────────────────
    ing_dash -->|":8080"| dashboard
    ing_api -->|":8000"| query_api
    ing_reg -->|":8000"| symbol_registry
    ing_trade -->|":8000"| trading_engine
    ing_superset -->|":8088"| superset
    ing_trino -->|":8080"| trino

    %% ── Dashboard → Backend APIs ──────────────────────────────────
    dashboard -.->|"/api/ proxy"| query_api
    dashboard -.->|"/registry/ proxy"| symbol_registry
    dashboard -.->|"/risk/ proxy"| risk_engine

    %% ── Pipeline data flow (via Redis queues) ─────────────────────
    scheduler -->|"enqueue jobs"| redis
    ingestion -->|"stonks:queue:parsing"| redis
    parser -->|"stonks:queue:extraction"| redis
    extractor -->|"stonks:queue:aggregation"| redis
    aggregation -->|"stonks:queue:recommendation"| redis
    recommendation -->|"stonks:queue:trading_decisions"| redis
    trading_engine -->|"stonks:queue:broker_orders"| redis
    broker_adapter -->|"read orders"| redis
    lake_publisher -->|"stonks:queue:lake_publish"| redis

    %% ── External service connections ──────────────────────────────
    scheduler --> postgres
    scheduler --> redis
    ingestion --> postgres
    ingestion --> redis
    ingestion --> minio
    parser --> postgres
    parser --> redis
    extractor --> postgres
    extractor --> redis
    extractor --> ollama
    aggregation --> postgres
    aggregation --> redis
    recommendation --> postgres
    recommendation --> redis
    trading_engine --> postgres
    trading_engine --> redis
    risk_engine --> postgres
    broker_adapter --> postgres
    broker_adapter --> redis
    lake_publisher --> postgres
    lake_publisher --> minio
    query_api --> postgres
    query_api --> redis
    query_api --> minio
    symbol_registry --> postgres

    %% ── Analytics plane connections ───────────────────────────────
    lake_publisher -->|"Parquet → s3a://stonks-lakehouse"| minio
    hive_metastore -->|"s3a:// catalog"| minio
    trino -->|"thrift://hive-metastore:9083"| hive_metastore
    superset -->|"trino:8080"| trino
    query_api -->|"trino:8080"| trino
    superset --> postgres
    superset --> redis

    %% ── Trading tier external egress ──────────────────────────────
    trading_engine -->|"HTTPS :443<br/>Alpaca API"| internet
    trading_engine -->|"SMTP :587<br/>Gmail notifications"| internet
    broker_adapter -->|"HTTPS :443<br/>Alpaca API"| internet
    ingestion -->|"HTTPS :443<br/>Polygon.io / News APIs"| internet

    %% ── Secret consumption ────────────────────────────────────────
    sec_core -.-> query_api
    sec_core -.-> symbol_registry
    sec_core -.-> scheduler
    sec_core -.-> ingestion
    sec_core -.-> parser
    sec_core -.-> extractor
    sec_core -.-> aggregation
    sec_core -.-> recommendation
    sec_core -.-> trading_engine
    sec_core -.-> risk_engine
    sec_core -.-> broker_adapter
    sec_core -.-> lake_publisher
    sec_core -.-> hive_metastore
    sec_core -.-> trino
    sec_core -.-> superset

    sec_broker -.-> ingestion
    sec_broker -.-> trading_engine
    sec_broker -.-> risk_engine
    sec_broker -.-> broker_adapter

    sec_market -.-> ingestion
    sec_market -.-> query_api

    sec_gmail -.-> trading_engine

    sec_dashboard -.-> superset

    configmap -.-> query_api
    configmap -.-> symbol_registry
    configmap -.-> scheduler
    configmap -.-> ingestion
    configmap -.-> parser
    configmap -.-> extractor
    configmap -.-> aggregation
    configmap -.-> recommendation
    configmap -.-> trading_engine
    configmap -.-> risk_engine
    configmap -.-> broker_adapter
    configmap -.-> lake_publisher
    configmap -.-> superset

    %% ── Styles ────────────────────────────────────────────────────
    classDef apiSvc fill:#4a90d9,stroke:#2c5f8a,color:#fff
    classDef frontendSvc fill:#50c878,stroke:#2e7d46,color:#fff
    classDef tradingSvc fill:#e8a838,stroke:#b07d1a,color:#fff
    classDef processSvc fill:#9b59b6,stroke:#6c3483,color:#fff
    classDef orchSvc fill:#1abc9c,stroke:#148f77,color:#fff
    classDef ingestionSvc fill:#e67e22,stroke:#bf6516,color:#fff
    classDef analyticsSvc fill:#e74c3c,stroke:#a93226,color:#fff
    classDef supersetSvc fill:#c0392b,stroke:#96281b,color:#fff
    classDef extSvc fill:#95a5a6,stroke:#717d7e,color:#fff
    classDef secretSvc fill:#f5f5dc,stroke:#999,color:#333
    classDef configSvc fill:#dfe6e9,stroke:#999,color:#333

    class query_api,symbol_registry apiSvc
    class dashboard frontendSvc
    class trading_engine,risk_engine,broker_adapter tradingSvc
    class scheduler orchSvc
    class ingestion ingestionSvc
    class parser,extractor,aggregation,recommendation processSvc
    class lake_publisher,hive_metastore,trino analyticsSvc
    class superset supersetSvc
    class postgres,redis,minio,ollama extSvc
    class sec_core,sec_broker,sec_market,sec_gmail,sec_dashboard secretSvc
    class configmap configSvc
```

## Network Policy Boundaries

The Helm chart deploys a **default-deny-ingress** policy that blocks all inbound traffic to pods in the `stonks-oracle` namespace. Each service that needs inbound connections has an explicit allow policy:

```mermaid
graph LR
    subgraph netpol ["Network Policies — stonks-oracle namespace"]
        direction TB

        deny["🔒 default-deny-ingress<br/><i>Blocks ALL ingress to all pods</i>"]

        subgraph allows ["Explicit Allow Rules"]
            direction TB

            np_dash["allow-dashboard-ingress<br/>dashboard :8080<br/>← kube-system (Traefik)"]

            np_api["allow-query-api-ingress<br/>query-api :8000<br/>← kube-system (Traefik)<br/>← dashboard pod"]

            np_reg["allow-symbol-registry-ingress<br/>symbol-registry :8000<br/>← kube-system (Traefik)<br/>← dashboard pod"]

            np_trade["allow-trading-engine-ingress<br/>trading-engine :8000<br/>← kube-system (Traefik)<br/>← query-api pod<br/>← dashboard pod<br/><i>Egress: PostgreSQL :5432,</i><br/><i>Redis :6379, HTTPS :443, SMTP :587</i>"]

            np_risk["allow-risk-engine-ingress<br/>risk-engine :8000<br/>← broker-adapter pod<br/>← query-api pod<br/>← dashboard pod"]

            np_superset["allow-superset-ingress<br/>superset :8088<br/>← kube-system (Traefik)"]

            np_trino["allow-trino-ingress<br/>trino :8080<br/>← superset pod<br/>← query-api pod<br/>← kube-system (Traefik)"]

            np_hive["allow-hive-metastore-ingress<br/>hive-metastore :9083<br/>← trino pod<br/>← lake-publisher pod"]

            np_broker["deny-broker-adapter-ingress<br/>broker-adapter<br/><i>No inbound traffic allowed</i>"]
        end
    end

    style deny fill:#e74c3c,stroke:#c0392b,color:#fff
    style np_broker fill:#e74c3c,stroke:#c0392b,color:#fff
    style np_dash fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_api fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_reg fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_trade fill:#f39c12,stroke:#d68910,color:#fff
    style np_risk fill:#f39c12,stroke:#d68910,color:#fff
    style np_superset fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_trino fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_hive fill:#3498db,stroke:#2980b9,color:#fff
```

### Services Without Ingress Policies (Pipeline Workers)

The following services have **no inbound network policy** — they are queue-driven workers that only make outbound connections to PostgreSQL, Redis, MinIO, and Ollama. The default-deny-ingress policy blocks any unsolicited inbound traffic:

| Service | Tier | Behavior |
|---------|------|----------|
| scheduler | orchestration | Polls DB, enqueues to Redis. Runs migrations + seed + backfill as init containers |
| ingestion | ingestion | Reads from `stonks:queue:ingestion`, writes to DB/MinIO/Redis. Egress to Polygon.io/News APIs |
| parser | processing | Reads from `stonks:queue:parsing`, writes to DB/Redis |
| extractor | processing | Reads from `stonks:queue:extraction`, calls Ollama, writes to DB/Redis |
| aggregation | processing | Reads from `stonks:queue:aggregation`, writes to DB/Redis |
| recommendation | processing | Reads from `stonks:queue:recommendation`, writes to DB/Redis |
| lake-publisher | analytics | Reads from `stonks:queue:lake_publish`, writes Parquet to MinIO |

## Service Tier Summary

| Tier | Services | Ingress? | Replicas | Pipeline-Gated? | Notes |
|------|----------|----------|----------|-----------------|-------|
| **api** | query-api, symbol-registry | Yes (Traefik) | 1 each | No | FastAPI, readiness probes on `/docs` |
| **frontend** | dashboard | Yes (Traefik) | 1 | No | nginx-unprivileged on :8080, proxies to API services |
| **trading** | trading-engine, risk-engine, broker-adapter | trading-engine: Yes; risk-engine: internal only; broker-adapter: denied | 1 each | broker-adapter only | trading-engine has egress to Alpaca + Gmail |
| **orchestration** | scheduler | No | 1 | Yes | Runs DB migrations + seed + backfill as init containers |
| **ingestion** | ingestion | No | 1 | Yes | Fetches from external APIs (Polygon.io, news, filings) |
| **processing** | parser, extractor, aggregation, recommendation | No | 2, 1, 4, 1 | Yes | Queue-driven pipeline workers |
| **analytics** | lake-publisher, trino, hive-metastore | trino: Yes (Traefik); others: No | 1 each | lake-publisher only | trino + hive-metastore gated by `trino.enabled` / `hiveMetastore.enabled` |
| **dashboard** (Superset) | superset | Yes (Traefik) | 1 | No | Gated by `superset.enabled`, custom image with trino + psycopg2 drivers |

## Secret Consumption Map

| Secret | Keys | Consumers |
|--------|------|-----------|
| `stonks-core-secrets` | POSTGRES_PASSWORD, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, REDIS_PASSWORD | All 13 app services + hive-metastore (init), trino (init), superset |
| `stonks-broker-secrets` | BROKER_API_KEY, BROKER_API_SECRET, BROKER_BASE_URL | ingestion, trading-engine, risk-engine, broker-adapter |
| `stonks-market-secrets` | MARKET_DATA_API_KEY | ingestion, query-api |
| `stonks-gmail-secrets` | GMAIL_SENDER, GMAIL_RECIPIENT, GMAIL_APP_PASSWORD | trading-engine |
| `stonks-dashboard-secrets` | SUPERSET_SECRET_KEY, SUPERSET_ADMIN_PASSWORD | superset |

## Pipeline Toggle

Setting `pipelineEnabled: false` in `values.yaml` scales all services with `pipeline: true` to 0 replicas. This affects:

- scheduler, ingestion, parser, extractor, aggregation, recommendation, broker-adapter, lake-publisher

API-tier services (query-api, symbol-registry), trading-tier services (trading-engine, risk-engine), analytics services (trino, hive-metastore, superset), and the dashboard always run regardless of this toggle.

## External Cluster Services

These services run outside the `stonks-oracle` namespace and are referenced via cross-namespace DNS:

| Service | Namespace | DNS | Port | Notes |
|---------|-----------|-----|------|-------|
| PostgreSQL | `postgresql-service` | `postgresql-rw.postgresql-service.svc.cluster.local` | 5432 | CloudNativePG managed |
| Redis | `redis-service` | `redis-master.redis-service.svc.cluster.local` | 6379 | Password in `stonks-core-secrets` |
| MinIO | `minio-service` | `minio.minio-service.svc.cluster.local` | 80 | S3-compatible object store |
| Ollama | `ollama-service` | `ollama.ollama-service.svc.cluster.local` | 11434 | LLM inference, GPU: 4070 Ti Super 16GB |

## Analytics Plane

The analytics stack runs within the `stonks-oracle` namespace:

1. **Lake Publisher** writes Parquet fact tables to MinIO at `s3a://stonks-lakehouse/warehouse`. Pipeline-gated — scales to 0 when `pipelineEnabled: false`.
2. **Hive Metastore** (Apache Hive 4.0.0) manages table metadata, backed by embedded Derby DB with a PVC (`hive-metastore-data`) for persistence. Connects to MinIO for S3A filesystem access. Gated by `hiveMetastore.enabled`.
3. **Trino** queries the lakehouse via Hive Metastore (`thrift://hive-metastore:9083`). Exposes two catalogs: `lakehouse` (Hive connector) and `iceberg` (Iceberg connector). Both connect to MinIO for data access. Gated by `trino.enabled`. Readiness probe on `/v1/info`.
4. **Superset** connects to Trino for lakehouse queries and to PostgreSQL for its metadata DB. Uses Redis for caching. Exposed externally via Traefik ingress. Gated by `superset.enabled`. Uses custom image (`registry.celestium.life/stonks-oracle/superset:latest`) with trino + psycopg2 drivers. PVC (`superset-data`) for persistence.

## Ingress Routes

All ingress resources use the `traefik` IngressClass with TLS certificates issued by the `ca-issuer` ClusterIssuer:

| Domain | Backend Service | Port | TLS Secret |
|--------|----------------|------|------------|
| `stonks.celestium.life` | dashboard | 8080 | `stonks-dashboard-tls` |
| `stonks-api.celestium.life` | query-api | 8000 | `stonks-api-tls` |
| `stonks-registry.celestium.life` | symbol-registry | 8000 | `stonks-registry-tls` |
| `stonks-trading.celestium.life` | trading-engine | 8000 | `stonks-trading-tls` |
| `stonks-dash.celestium.life` | superset | 8088 | `stonks-dash-tls` |
| `stonks-trino.celestium.life` | trino | 8080 | `stonks-trino-tls` |

## Deployment Stages

The Helm chart supports multiple deployment stages via value override files:

| Stage | Override File | Namespace | Key Differences |
|-------|--------------|-----------|-----------------|
| **Production** | `values.yaml` (base) | `stonks-oracle` | Full analytics stack, all services |
| **Paper** | `values-paper.yaml` | `stonks-oracle` | `BROKER_MODE=paper`, `DEPLOY_STAGE=paper`, separate DB (`stonks_paper`), Redis DB 2, paper-specific ingress hostnames |
| **Beta** | `values-beta.yaml` | `stonks-oracle-beta` | `DEPLOY_STAGE=beta`, `LOG_LEVEL=DEBUG`, separate DB (`stonks_beta`), Redis DB 1, analytics stack disabled, beta-specific ingress hostnames |