stonks-oracle/docs/architecture-kubernetes.md

Kubernetes Architecture — Stonks Oracle

This document describes the Kubernetes deployment topology for Stonks Oracle, derived from the Helm chart at infra/helm/stonks-oracle/.

All application workloads deploy to the stonks-oracle namespace. External cluster services (PostgreSQL, Redis, MinIO, Ollama) run in their own namespaces and are referenced via cross-namespace DNS.

Deployment Diagram

graph TB
    %% ── External traffic ──────────────────────────────────────────
    internet((Internet))

    subgraph traefik ["kube-system (Traefik Ingress Controller)"]
        direction LR
        ing_dash["stonks.celestium.life"]
        ing_api["stonks-api.celestium.life"]
        ing_reg["stonks-registry.celestium.life"]
        ing_trade["stonks-trading.celestium.life"]
        ing_superset["stonks-dash.celestium.life"]
        ing_trino["stonks-trino.celestium.life"]
    end

    internet --> traefik

    %% ── stonks-oracle namespace ───────────────────────────────────
    subgraph ns ["stonks-oracle namespace"]
        direction TB

        %% ── API Tier (ingress-facing) ─────────────────────────────
        subgraph api_tier ["API Tier"]
            direction LR
            query_api["query-api<br/><i>Deployment (1 replica)</i><br/>:8000"]
            symbol_registry["symbol-registry<br/><i>Deployment (1 replica)</i><br/>:8000"]
        end

        %% ── Frontend Tier ─────────────────────────────────────────
        subgraph frontend_tier ["Frontend Tier"]
            dashboard["dashboard<br/><i>Deployment (1 replica)</i><br/>:8080<br/><i>nginx-unprivileged</i>"]
        end

        %% ── Trading Tier ──────────────────────────────────────────
        subgraph trading_tier ["Trading Tier"]
            direction LR
            trading_engine["trading-engine<br/><i>Deployment (1 replica)</i><br/>:8000"]
            risk_engine["risk-engine<br/><i>Deployment (1 replica)</i><br/>:8000"]
            broker_adapter["broker-adapter<br/><i>Deployment (1 replica)</i><br/><i>queue-driven worker</i>"]
        end

        %% ── Orchestration Tier ────────────────────────────────────
        subgraph orchestration_tier ["Orchestration Tier"]
            scheduler["scheduler<br/><i>Deployment (1 replica)</i><br/><i>runs migrations + seed</i>"]
        end

        %% ── Processing Tier (pipeline workers) ────────────────────
        subgraph processing_tier ["Processing Tier (pipeline workers)"]
            direction LR
            ingestion["ingestion<br/><i>Deployment (2 replicas)</i>"]
            parser["parser<br/><i>Deployment (2 replicas)</i>"]
            extractor["extractor<br/><i>Deployment (1 replica)</i>"]
            aggregation["aggregation<br/><i>Deployment (4 replicas)</i>"]
            recommendation["recommendation<br/><i>Deployment (1 replica)</i>"]
        end

        %% ── Analytics Tier ────────────────────────────────────────
        subgraph analytics_tier ["Analytics Tier"]
            direction LR
            lake_publisher["lake-publisher<br/><i>Deployment (1 replica)</i><br/><i>queue-driven worker</i>"]
            hive_metastore["hive-metastore<br/><i>Deployment (1 replica)</i><br/>:9083<br/><i>apache/hive:4.0.0</i>"]
            trino["trino<br/><i>Deployment (1 replica)</i><br/>:8080<br/><i>trinodb/trino:latest</i>"]
            superset["superset<br/><i>Deployment (1 replica)</i><br/>:8088<br/><i>custom image</i>"]
        end

        %% ── Helm Secrets ──────────────────────────────────────────
        subgraph secrets_block ["Helm-Managed Secrets"]
            direction LR
            sec_core["stonks-core-secrets<br/><i>POSTGRES_PASSWORD</i><br/><i>MINIO_ACCESS_KEY</i><br/><i>MINIO_SECRET_KEY</i><br/><i>REDIS_PASSWORD</i>"]
            sec_broker["stonks-broker-secrets<br/><i>BROKER_API_KEY</i><br/><i>BROKER_API_SECRET</i><br/><i>BROKER_BASE_URL</i>"]
            sec_market["stonks-market-secrets<br/><i>MARKET_DATA_API_KEY</i>"]
            sec_gmail["stonks-gmail-secrets<br/><i>GMAIL_SENDER</i><br/><i>GMAIL_RECIPIENT</i><br/><i>GMAIL_APP_PASSWORD</i>"]
            sec_dashboard["stonks-dashboard-secrets<br/><i>SUPERSET_SECRET_KEY</i><br/><i>SUPERSET_ADMIN_PASSWORD</i>"]
        end

        %% ── ConfigMap ─────────────────────────────────────────────
        configmap["stonks-config<br/><i>ConfigMap</i><br/><i>All env vars from values.yaml config block</i>"]
    end

    %% ── External Cluster Services ─────────────────────────────────
    subgraph pg_ns ["postgresql-service namespace"]
        postgres[("PostgreSQL<br/>postgresql-rw:5432")]
    end

    subgraph redis_ns ["redis-service namespace"]
        redis[("Redis<br/>redis-master:6379")]
    end

    subgraph minio_ns ["minio-service namespace"]
        minio[("MinIO<br/>minio:80")]
    end

    subgraph ollama_ns ["ollama-service namespace"]
        ollama[("Ollama<br/>ollama:11434<br/><i>GPU: 4070 Ti Super</i>")]
    end

    %% ── Ingress Routes ────────────────────────────────────────────
    ing_dash -->|":8080"| dashboard
    ing_api -->|":8000"| query_api
    ing_reg -->|":8000"| symbol_registry
    ing_trade -->|":8000"| trading_engine
    ing_superset -->|":8088"| superset
    ing_trino -->|":8080"| trino

    %% ── Dashboard → Backend APIs ──────────────────────────────────
    dashboard -.->|"/api/ proxy"| query_api
    dashboard -.->|"/registry/ proxy"| symbol_registry
    dashboard -.->|"/risk/ proxy"| risk_engine

    %% ── Pipeline data flow (via Redis queues) ─────────────────────
    scheduler -->|"enqueue jobs"| redis
    ingestion -->|"stonks:queue:parsing"| redis
    parser -->|"stonks:queue:extraction"| redis
    extractor -->|"stonks:queue:aggregation"| redis
    aggregation -->|"stonks:queue:recommendation"| redis
    recommendation -->|"stonks:queue:trading_decisions"| redis
    trading_engine -->|"stonks:queue:broker_orders"| redis
    broker_adapter -->|"read orders"| redis
    lake_publisher -->|"stonks:queue:lake_publish"| redis

    %% ── External service connections ──────────────────────────────
    scheduler --> postgres
    scheduler --> redis
    ingestion --> postgres
    ingestion --> redis
    ingestion --> minio
    parser --> postgres
    parser --> redis
    extractor --> postgres
    extractor --> redis
    extractor --> ollama
    aggregation --> postgres
    aggregation --> redis
    recommendation --> postgres
    recommendation --> redis
    trading_engine --> postgres
    trading_engine --> redis
    risk_engine --> postgres
    broker_adapter --> postgres
    broker_adapter --> redis
    lake_publisher --> postgres
    lake_publisher --> minio
    query_api --> postgres
    query_api --> redis
    query_api --> minio
    symbol_registry --> postgres

    %% ── Analytics plane connections ───────────────────────────────
    lake_publisher -->|"Parquet → s3a://stonks-lakehouse"| minio
    hive_metastore -->|"s3a:// catalog"| minio
    trino -->|"thrift://hive-metastore:9083"| hive_metastore
    superset -->|"trino:8080"| trino
    query_api -->|"trino:8080"| trino
    superset --> postgres
    superset --> redis

    %% ── Trading tier external egress ──────────────────────────────
    trading_engine -->|"HTTPS :443<br/>Alpaca API"| internet
    trading_engine -->|"SMTP :587<br/>Gmail notifications"| internet
    broker_adapter -->|"HTTPS :443<br/>Alpaca API"| internet
    ingestion -->|"HTTPS :443<br/>Polygon.io / News APIs"| internet

    %% ── Secret consumption ────────────────────────────────────────
    sec_core -.-> query_api
    sec_core -.-> symbol_registry
    sec_core -.-> scheduler
    sec_core -.-> ingestion
    sec_core -.-> parser
    sec_core -.-> extractor
    sec_core -.-> aggregation
    sec_core -.-> recommendation
    sec_core -.-> trading_engine
    sec_core -.-> risk_engine
    sec_core -.-> broker_adapter
    sec_core -.-> lake_publisher
    sec_core -.-> hive_metastore
    sec_core -.-> trino
    sec_core -.-> superset

    sec_broker -.-> ingestion
    sec_broker -.-> trading_engine
    sec_broker -.-> risk_engine
    sec_broker -.-> broker_adapter

    sec_market -.-> ingestion

    sec_gmail -.-> trading_engine

    sec_dashboard -.-> superset

    configmap -.-> query_api
    configmap -.-> symbol_registry
    configmap -.-> scheduler
    configmap -.-> ingestion
    configmap -.-> parser
    configmap -.-> extractor
    configmap -.-> aggregation
    configmap -.-> recommendation
    configmap -.-> trading_engine
    configmap -.-> risk_engine
    configmap -.-> broker_adapter
    configmap -.-> lake_publisher
    configmap -.-> superset

    %% ── Styles ────────────────────────────────────────────────────
    classDef apiSvc fill:#4a90d9,stroke:#2c5f8a,color:#fff
    classDef frontendSvc fill:#50c878,stroke:#2e7d46,color:#fff
    classDef tradingSvc fill:#e8a838,stroke:#b07d1a,color:#fff
    classDef processSvc fill:#9b59b6,stroke:#6c3483,color:#fff
    classDef orchSvc fill:#1abc9c,stroke:#148f77,color:#fff
    classDef analyticsSvc fill:#e74c3c,stroke:#a93226,color:#fff
    classDef extSvc fill:#95a5a6,stroke:#717d7e,color:#fff
    classDef secretSvc fill:#f5f5dc,stroke:#999,color:#333
    classDef configSvc fill:#dfe6e9,stroke:#999,color:#333

    class query_api,symbol_registry apiSvc
    class dashboard frontendSvc
    class trading_engine,risk_engine,broker_adapter tradingSvc
    class scheduler orchSvc
    class ingestion,parser,extractor,aggregation,recommendation processSvc
    class lake_publisher,hive_metastore,trino,superset analyticsSvc
    class postgres,redis,minio,ollama extSvc
    class sec_core,sec_broker,sec_market,sec_gmail,sec_dashboard secretSvc
    class configmap configSvc

Network Policy Boundaries

The Helm chart deploys a default-deny-ingress policy that blocks all inbound traffic to pods in the stonks-oracle namespace. Each service that needs inbound connections has an explicit allow policy:

graph LR
    subgraph netpol ["Network Policies — stonks-oracle namespace"]
        direction TB

        deny["🔒 default-deny-ingress<br/><i>Blocks ALL ingress to all pods</i>"]

        subgraph allows ["Explicit Allow Rules"]
            direction TB

            np_dash["allow-dashboard-ingress<br/>dashboard :8080<br/>← kube-system (Traefik)"]

            np_api["allow-query-api-ingress<br/>query-api :8000<br/>← kube-system (Traefik)<br/>← dashboard pod"]

            np_reg["allow-symbol-registry-ingress<br/>symbol-registry :8000<br/>← kube-system (Traefik)<br/>← dashboard pod"]

            np_trade["allow-trading-engine-ingress<br/>trading-engine :8000<br/>← kube-system (Traefik)<br/>← query-api pod<br/>← dashboard pod<br/><i>Egress: PostgreSQL :5432,</i><br/><i>Redis :6379, HTTPS :443, SMTP :587</i>"]

            np_risk["allow-risk-engine-ingress<br/>risk-engine :8000<br/>← broker-adapter pod<br/>← query-api pod<br/>← dashboard pod"]

            np_superset["allow-superset-ingress<br/>superset :8088<br/>← kube-system (Traefik)"]

            np_trino["allow-trino-ingress<br/>trino :8080<br/>← superset pod<br/>← query-api pod<br/>← kube-system (Traefik)"]

            np_hive["allow-hive-metastore-ingress<br/>hive-metastore :9083<br/>← trino pod<br/>← lake-publisher pod"]

            np_broker["deny-broker-adapter-ingress<br/>broker-adapter<br/><i>No inbound traffic allowed</i>"]
        end
    end

    style deny fill:#e74c3c,stroke:#c0392b,color:#fff
    style np_broker fill:#e74c3c,stroke:#c0392b,color:#fff
    style np_dash fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_api fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_reg fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_trade fill:#f39c12,stroke:#d68910,color:#fff
    style np_risk fill:#f39c12,stroke:#d68910,color:#fff
    style np_superset fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_trino fill:#2ecc71,stroke:#27ae60,color:#fff
    style np_hive fill:#3498db,stroke:#2980b9,color:#fff

Services Without Ingress Policies (Pipeline Workers)

The following services have no inbound network policy — they are queue-driven workers that only make outbound connections to PostgreSQL, Redis, MinIO, and Ollama. The default-deny-ingress policy blocks any unsolicited inbound traffic:

Service	Tier	Behavior
scheduler	orchestration	Polls DB, enqueues to Redis
ingestion	processing	Reads from stonks:queue:ingestion, writes to DB/MinIO/Redis
parser	processing	Reads from stonks:queue:parsing, writes to DB/Redis
extractor	processing	Reads from stonks:queue:extraction, calls Ollama, writes to DB/Redis
aggregation	processing	Reads from stonks:queue:aggregation, writes to DB/Redis
recommendation	processing	Reads from stonks:queue:recommendation, writes to DB/Redis
lake-publisher	analytics	Reads from stonks:queue:lake_publish, writes Parquet to MinIO

Service Tier Summary

Tier	Services	Ingress?	Replicas	Notes
api	query-api, symbol-registry	Yes (Traefik)	1 each	FastAPI, readiness probes on /docs
frontend	dashboard	Yes (Traefik)	1	nginx-unprivileged on :8080, proxies to API services
trading	trading-engine, risk-engine, broker-adapter	trading-engine: Yes; risk-engine: internal only; broker-adapter: denied	1 each	trading-engine has egress to Alpaca + Gmail
orchestration	scheduler	No	1	Runs DB migrations + seed as init containers
processing	ingestion, parser, extractor, aggregation, recommendation	No	2, 2, 1, 4, 1	Pipeline-gated by pipelineEnabled toggle
analytics	lake-publisher, trino, hive-metastore, superset	trino + superset: Yes; others: No	1 each	lake-publisher is pipeline-gated

Secret Consumption Map

Secret	Keys	Consumers
stonks-core-secrets	POSTGRES_PASSWORD, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, REDIS_PASSWORD	All 13 app services + hive-metastore, trino, superset
stonks-broker-secrets	BROKER_API_KEY, BROKER_API_SECRET, BROKER_BASE_URL	ingestion, trading-engine, risk-engine, broker-adapter
stonks-market-secrets	MARKET_DATA_API_KEY	ingestion
stonks-gmail-secrets	GMAIL_SENDER, GMAIL_RECIPIENT, GMAIL_APP_PASSWORD	trading-engine
stonks-dashboard-secrets	SUPERSET_SECRET_KEY, SUPERSET_ADMIN_PASSWORD	superset

Pipeline Toggle

Setting pipelineEnabled: false in values.yaml scales all services with pipeline: true to 0 replicas. This affects:

• scheduler, ingestion, parser, extractor, aggregation, recommendation, broker-adapter, lake-publisher

API-tier services (query-api, symbol-registry), trading-tier services (trading-engine, risk-engine), analytics services (trino, hive-metastore, superset), and the dashboard always run regardless of this toggle.

External Cluster Services

These services run outside the stonks-oracle namespace and are referenced via cross-namespace DNS:

Service	Namespace	DNS	Port	Notes
PostgreSQL	postgresql-service	postgresql-rw.postgresql-service.svc.cluster.local	5432	CloudNativePG managed
Redis	redis-service	redis-master.redis-service.svc.cluster.local	6379	Password in stonks-core-secrets
MinIO	minio-service	minio.minio-service.svc.cluster.local	80	S3-compatible object store
Ollama	ollama-service	ollama.ollama-service.svc.cluster.local	11434	LLM inference, GPU: 4070 Ti Super 16GB

Analytics Plane

The analytics stack runs within the stonks-oracle namespace:

1. Lake Publisher writes Parquet fact tables to MinIO at s3a://stonks-lakehouse/warehouse
2. Hive Metastore (Apache Hive 4.0.0) manages table metadata, backed by embedded Derby DB with a PVC for persistence. Connects to MinIO for S3A filesystem access.
3. Trino queries the lakehouse via Hive Metastore (thrift://hive-metastore:9083). Exposes two catalogs: lakehouse (Hive connector) and iceberg (Iceberg connector). Both connect to MinIO for data access.
4. Superset connects to Trino for lakehouse queries and to PostgreSQL for its metadata DB. Uses Redis for caching. Exposed externally via Traefik ingress.

Ingress Routes

All ingress resources use the traefik IngressClass with TLS certificates issued by the ca-issuer ClusterIssuer:

Domain	Backend Service	Port	TLS Secret
stonks.celestium.life	dashboard	8080	stonks-dashboard-tls
stonks-api.celestium.life	query-api	8000	stonks-api-tls
stonks-registry.celestium.life	symbol-registry	8000	stonks-registry-tls
stonks-trading.celestium.life	trading-engine	8000	stonks-trading-tls
stonks-dash.celestium.life	superset	8088	stonks-dash-tls
stonks-trino.celestium.life	trino	8080	stonks-trino-tls