# Docker Compose Architecture — Stonks Oracle

This document describes the Docker Compose deployment topology for Stonks Oracle, derived from the `docker-compose.yml` file at the repository root.

All containers run on a single Docker network created by Compose. Infrastructure services (PostgreSQL, Redis, MinIO, Ollama, Trino, Hive Metastore, Superset) start first, and application services wait for their dependencies via `depends_on` with health check conditions.

## Container Topology Diagram

```mermaid
graph TB
    %% ── Host machine ──────────────────────────────────────────────
    host((Host Machine))

    %% ── .env file ─────────────────────────────────────────────────
    envfile[".env file<br/><i>MARKET_DATA_API_KEY</i><br/><i>BROKER_API_KEY</i><br/><i>BROKER_API_SECRET</i><br/><i>BROKER_BASE_URL</i>"]

    %% ── Docker Compose default network ────────────────────────────
    subgraph network ["Docker Compose Network (default)"]
        direction TB

        %% ── Infrastructure Containers ─────────────────────────────
        subgraph infra ["Infrastructure Containers"]
            direction LR
            postgres[("postgres<br/><i>postgres:16-alpine</i><br/>host :5432 → :5432")]
            redis[("redis<br/><i>redis:7-alpine</i><br/>host :6379 → :6379")]
            minio[("minio<br/><i>minio/minio:latest</i><br/>host :9000 → :9000<br/>host :9001 → :9001")]
            ollama[("ollama<br/><i>ollama/ollama:latest</i><br/>host :11434 → :11434")]
        end

        subgraph infra_init ["Infrastructure Init"]
            minio_init["minio-init<br/><i>minio/mc:latest</i><br/><i>Creates buckets on startup</i>"]
        end

        subgraph analytics ["Analytics Containers"]
            direction LR
            hive_metastore["hive-metastore<br/><i>apache/hive:4.0.0</i><br/>host :9083 → :9083"]
            trino["trino<br/><i>trinodb/trino:latest</i><br/>host :8080 → :8080"]
            superset["superset<br/><i>apache/superset:latest</i><br/>host :8088 → :8088"]
        end

        %% ── Application Containers ────────────────────────────────

        subgraph api_tier ["API Tier"]
            direction LR
            query_api["query-api<br/><i>docker/Dockerfile</i><br/><i>uvicorn services.api.app</i><br/>host :8004 → :8000"]
            symbol_registry["symbol-registry<br/><i>docker/Dockerfile</i><br/><i>uvicorn services.symbol_registry.app</i><br/>host :8001 → :8000"]
        end

        subgraph frontend_tier ["Frontend Tier"]
            dashboard["dashboard<br/><i>frontend/Dockerfile</i><br/><i>nginx on :8080</i><br/>host :3000 → :8080"]
        end

        subgraph trading_tier ["Trading Tier"]
            direction LR
            trading_engine["trading-engine<br/><i>docker/Dockerfile</i><br/><i>uvicorn services.trading.app</i><br/>host :8002 → :8000"]
            risk_engine["risk-engine<br/><i>docker/Dockerfile</i><br/><i>uvicorn services.risk.app</i><br/>host :8003 → :8000<br/><i>alias: risk</i>"]
            broker_adapter["broker-adapter<br/><i>docker/Dockerfile</i><br/><i>python -m services.adapters.broker_service</i><br/><i>no host port</i>"]
        end

        subgraph orchestration_tier ["Orchestration Tier"]
            scheduler["scheduler<br/><i>docker/Dockerfile.scheduler</i><br/><i>no host port</i>"]
        end

        subgraph processing_tier ["Processing Tier (pipeline workers)"]
            direction LR
            ingestion["ingestion<br/><i>docker/Dockerfile</i><br/><i>python -m services.ingestion.worker</i><br/><i>no host port</i>"]
            parser["parser<br/><i>docker/Dockerfile</i><br/><i>python -m services.parser.worker</i><br/><i>no host port</i>"]
            extractor["extractor<br/><i>docker/Dockerfile</i><br/><i>python -m services.extractor.main</i><br/><i>no host port</i>"]
            aggregation["aggregation<br/><i>docker/Dockerfile</i><br/><i>python -m services.aggregation.main</i><br/><i>no host port</i>"]
            recommendation["recommendation<br/><i>docker/Dockerfile</i><br/><i>python -m services.recommendation.main</i><br/><i>no host port</i>"]
        end

        subgraph analytics_worker ["Analytics Worker"]
            lake_publisher["lake-publisher<br/><i>docker/Dockerfile</i><br/><i>python -m services.lake_publisher.jobs</i><br/><i>no host port</i>"]
        end
    end

    %% ── Host port access ──────────────────────────────────────────
    host -->|":5432"| postgres
    host -->|":6379"| redis
    host -->|":9000 / :9001"| minio
    host -->|":11434"| ollama
    host -->|":8080"| trino
    host -->|":9083"| hive_metastore
    host -->|":8088"| superset
    host -->|":8001"| symbol_registry
    host -->|":8004"| query_api
    host -->|":8002"| trading_engine
    host -->|":8003"| risk_engine
    host -->|":3000"| dashboard

    %% ── .env injection ────────────────────────────────────────────
    envfile -.->|"env_file: .env"| ingestion
    envfile -.->|"env_file: .env"| broker_adapter
    envfile -.->|"env_file: .env"| trading_engine

    %% ── Styles ────────────────────────────────────────────────────
    classDef infraSvc fill:#95a5a6,stroke:#717d7e,color:#fff
    classDef analyticsSvc fill:#e74c3c,stroke:#a93226,color:#fff
    classDef apiSvc fill:#4a90d9,stroke:#2c5f8a,color:#fff
    classDef frontendSvc fill:#50c878,stroke:#2e7d46,color:#fff
    classDef tradingSvc fill:#e8a838,stroke:#b07d1a,color:#fff
    classDef orchSvc fill:#1abc9c,stroke:#148f77,color:#fff
    classDef processSvc fill:#9b59b6,stroke:#6c3483,color:#fff
    classDef initSvc fill:#bdc3c7,stroke:#7f8c8d,color:#333
    classDef envSvc fill:#f5f5dc,stroke:#999,color:#333

    class postgres,redis,minio,ollama infraSvc
    class hive_metastore,trino,superset,lake_publisher analyticsSvc
    class query_api,symbol_registry apiSvc
    class dashboard frontendSvc
    class trading_engine,risk_engine,broker_adapter tradingSvc
    class scheduler orchSvc
    class ingestion,parser,extractor,aggregation,recommendation processSvc
    class minio_init initSvc
    class envfile envSvc
```

## Dependency Graph

The following diagram shows `depends_on` relationships and health check conditions. Solid arrows indicate `condition: service_healthy` (the dependent waits for the health check to pass). Dashed arrows indicate `condition: service_started` (the dependent waits only for the container to start).

```mermaid
graph LR
    %% ── Infrastructure health checks ──────────────────────────────
    postgres[("postgres<br/><i>pg_isready -U stonks</i>")]
    redis[("redis<br/><i>redis-cli ping</i>")]
    minio[("minio<br/><i>mc ready local</i>")]
    ollama[("ollama<br/><i>no health check</i>")]

    %% ── Analytics dependencies ────────────────────────────────────
    hive["hive-metastore"] -->|started| minio
    trino["trino"] -->|started| minio
    trino -->|started| hive
    superset["superset"] -->|started| trino
    minio_init["minio-init"] -->|healthy| minio

    %% ── Application depends_on (healthy) ──────────────────────────
    scheduler["scheduler"] -->|healthy| postgres
    scheduler -->|healthy| redis

    symbol_registry["symbol-registry"] -->|healthy| postgres

    ingestion["ingestion"] -->|healthy| postgres
    ingestion -->|healthy| redis
    ingestion -->|healthy| minio

    parser["parser"] -->|healthy| postgres
    parser -->|healthy| redis

    extractor["extractor"] -->|healthy| postgres
    extractor -->|healthy| redis
    extractor -.->|started| ollama

    aggregation["aggregation"] -->|healthy| postgres
    aggregation -->|healthy| redis

    recommendation["recommendation"] -->|healthy| postgres
    recommendation -->|healthy| redis

    trading_engine["trading-engine"] -->|healthy| postgres
    trading_engine -->|healthy| redis

    risk_engine["risk-engine"] -->|healthy| postgres

    broker_adapter["broker-adapter"] -->|healthy| postgres
    broker_adapter -->|healthy| redis

    lake_publisher["lake-publisher"] -->|healthy| postgres
    lake_publisher -->|healthy| minio

    query_api["query-api"] -->|healthy| postgres
    query_api -->|healthy| redis
    query_api -->|healthy| minio

    dashboard["dashboard"] -->|healthy| query_api

    %% ── Styles ────────────────────────────────────────────────────
    classDef infraSvc fill:#95a5a6,stroke:#717d7e,color:#fff
    classDef appSvc fill:#4a90d9,stroke:#2c5f8a,color:#fff
    classDef analyticsSvc fill:#e74c3c,stroke:#a93226,color:#fff
    classDef initSvc fill:#bdc3c7,stroke:#7f8c8d,color:#333

    class postgres,redis,minio,ollama infraSvc
    class scheduler,symbol_registry,ingestion,parser,extractor,aggregation,recommendation,trading_engine,risk_engine,broker_adapter,lake_publisher,query_api,dashboard appSvc
    class hive,trino,superset analyticsSvc
    class minio_init initSvc
```

## Named Volumes

Docker Compose defines five named volumes for persistent data:

```mermaid
graph LR
    pgdata["📦 pgdata"]
    miniodata["📦 miniodata"]
    ollama_models["📦 ollama_models"]
    hive_data["📦 hive_data"]
    superset_data["📦 superset_data"]

    pgdata -->|"/var/lib/postgresql/data"| postgres[("postgres")]
    miniodata -->|"/data"| minio[("minio")]
    ollama_models -->|"/root/.ollama"| ollama[("ollama")]
    hive_data -->|"/opt/hive/data"| hive["hive-metastore"]
    superset_data -->|"/app/superset_home"| superset["superset"]

    classDef volStyle fill:#f5f5dc,stroke:#999,color:#333
    classDef svcStyle fill:#95a5a6,stroke:#717d7e,color:#fff

    class pgdata,miniodata,ollama_models,hive_data,superset_data volStyle
    class postgres,minio,ollama,hive,superset svcStyle
```

| Volume | Mount Point | Container | Purpose |
|--------|-------------|-----------|---------|
| `pgdata` | `/var/lib/postgresql/data` | postgres | PostgreSQL database files |
| `miniodata` | `/data` | minio | MinIO object storage data |
| `ollama_models` | `/root/.ollama` | ollama | Downloaded LLM model weights |
| `hive_data` | `/opt/hive/data` | hive-metastore | Hive Metastore embedded Derby DB |
| `superset_data` | `/app/superset_home` | superset | Superset configuration and metadata |

### Bind Mounts

In addition to named volumes, several containers use bind mounts for configuration files:

| Host Path | Mount Point | Container | Mode |
|-----------|-------------|-----------|------|
| `./infra/migrations/` | `/docker-entrypoint-initdb.d` | postgres | rw (init scripts) |
| `./infra/trino/catalog/` | `/etc/trino/catalog` | trino | rw |
| `./infra/hive/core-site.xml` | `/opt/hive/conf/core-site.xml` | hive-metastore | ro |
| `./infra/hive/metastore-site.xml` | `/opt/hive/conf/metastore-site.xml` | hive-metastore | ro |

## Host Port Mappings

Services accessible from the host machine:

| Host Port | Container | Container Port | Service |
|-----------|-----------|----------------|---------|
| 5432 | postgres | 5432 | PostgreSQL database |
| 6379 | redis | 6379 | Redis cache and queues |
| 9000 | minio | 9000 | MinIO S3 API |
| 9001 | minio | 9001 | MinIO web console |
| 11434 | ollama | 11434 | Ollama LLM API |
| 8080 | trino | 8080 | Trino query engine |
| 9083 | hive-metastore | 9083 | Hive Metastore thrift |
| 8088 | superset | 8088 | Superset dashboard |
| 8001 | symbol-registry | 8000 | Symbol Registry API |
| 8002 | trading-engine | 8000 | Trading Engine API |
| 8003 | risk-engine | 8000 | Risk Engine API |
| 8004 | query-api | 8000 | Query API |
| 3000 | dashboard | 8080 | React dashboard (nginx) |

Services without host port mappings (internal only): scheduler, ingestion, parser, extractor, aggregation, recommendation, broker-adapter, lake-publisher, minio-init.

## Environment Configuration

### Shared Environment (`x-app-env` YAML anchor)

All 13 application services and the scheduler receive these environment variables via the `x-app-env` anchor:

| Variable | Value | Purpose |
|----------|-------|---------|
| `POSTGRES_HOST` | `postgres` | Docker Compose service name for PostgreSQL |
| `POSTGRES_PORT` | `5432` | PostgreSQL port |
| `POSTGRES_DB` | `stonks` | Database name |
| `POSTGRES_USER` | `stonks` | Database user |
| `POSTGRES_PASSWORD` | `stonks_dev` | Database password (dev default) |
| `REDIS_HOST` | `redis` | Docker Compose service name for Redis |
| `REDIS_PORT` | `6379` | Redis port |
| `MINIO_ENDPOINT` | `minio:9000` | Docker Compose service name for MinIO |
| `MINIO_ACCESS_KEY` | `minioadmin` | MinIO access key |
| `MINIO_SECRET_KEY` | `minioadmin` | MinIO secret key |
| `OLLAMA_BASE_URL` | `http://ollama:11434` | Docker Compose service name for Ollama |

### `.env` File (API Keys)

Three services load additional secrets from the `.env` file in the repository root via `env_file: .env`:

| Variable | Required By | Purpose |
|----------|-------------|---------|
| `MARKET_DATA_API_KEY` | ingestion | Polygon.io market data API key |
| `BROKER_API_KEY` | broker-adapter, trading-engine | Alpaca broker API key |
| `BROKER_API_SECRET` | broker-adapter, trading-engine | Alpaca broker API secret |
| `BROKER_BASE_URL` | broker-adapter, trading-engine | Alpaca API base URL (default: `https://paper-api.alpaca.markets`) |

## Health Check Summary

| Container | Health Check Command | Interval | Timeout | Retries | Start Period |
|-----------|---------------------|----------|---------|---------|--------------|
| postgres | `pg_isready -U stonks` | 5s | — | 5 | — |
| redis | `redis-cli ping` | 5s | — | 5 | — |
| minio | `mc ready local` | 5s | — | 5 | — |
| symbol-registry | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
| query-api | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
| trading-engine | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
| risk-engine | `curl -f http://localhost:8000/health` | 10s | 5s | 3 | 15s |
| dashboard | `curl -f http://localhost:8080/` | 10s | 5s | 3 | 10s |
| scheduler | `pgrep -f 'python -m services.scheduler.app'` | 10s | 5s | 3 | 15s |
| ingestion | `pgrep -f 'python -m services.ingestion.worker'` | 10s | 5s | 3 | 15s |
| parser | `pgrep -f 'python -m services.parser.worker'` | 10s | 5s | 3 | 15s |
| extractor | `pgrep -f 'python -m services.extractor.main'` | 10s | 5s | 3 | 15s |
| aggregation | `pgrep -f 'python -m services.aggregation.main'` | 10s | 5s | 3 | 15s |
| recommendation | `pgrep -f 'python -m services.recommendation.main'` | 10s | 5s | 3 | 15s |
| broker-adapter | `pgrep -f 'python -m services.adapters.broker_service'` | 10s | 5s | 3 | 15s |
| lake-publisher | `pgrep -f 'python -m services.lake_publisher.jobs'` | 10s | 5s | 3 | 15s |

Infrastructure services (ollama, trino, hive-metastore, superset) do not define health checks in docker-compose.yml. Application services that depend on ollama use `condition: service_started` instead of `condition: service_healthy`.

## Internal Network Connectivity

All containers share the default Docker Compose network. Services reference each other by their Compose service name as the hostname:

| Hostname | Resolved To | Used By |
|----------|-------------|---------|
| `postgres` | PostgreSQL container | All 13 app services, superset |
| `redis` | Redis container | scheduler, ingestion, parser, extractor, aggregation, recommendation, trading-engine, broker-adapter, query-api |
| `minio` | MinIO container | ingestion, lake-publisher, query-api (via `minio:9000`) |
| `ollama` | Ollama container | extractor (via `http://ollama:11434`) |
| `hive-metastore` | Hive Metastore container | trino (thrift://hive-metastore:9083) |
| `trino` | Trino container | superset (trino:8080) |
| `query-api` | Query API container | dashboard (nginx proxy upstream) |
| `risk` | risk-engine container (network alias) | trading-engine (risk evaluation calls) |