Celes Renata
376fcb4bb4
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
Prefixed with ⚕ for easy identification. Checks: duplicate snapshots, orphaned links, evidence count mismatches, contribution score sums, canonical key consistency, out-of-range values, unmatched snapshots, zero evidence rate, duplicate count mismatches, missing prices, outcome integrity, and pipeline health summary.
109 lines
7.0 KiB
Markdown
109 lines
7.0 KiB
Markdown
# Stonks Oracle — Project Context
|
|
|
|
## Overview
|
|
Stonks Oracle is a Kubernetes-native AI market intelligence and paper-trading platform.
|
|
Python monorepo with services under `services/`, infrastructure under `infra/`, lakehouse schemas under `lakehouse/`, frontend React dashboard under `frontend/`, and dashboards under `dashboards/`.
|
|
|
|
Three-layer signal aggregation engine:
|
|
1. **Company-specific signals** — document intelligence from news, filings, market data
|
|
2. **Macro signals** — global news interpolation, geopolitical event classification, exposure-based impact scoring
|
|
3. **Competitive signals** — historical pattern mining, cross-company signal propagation, competitor relationship management
|
|
|
|
## Tracked Universe
|
|
- 50 companies across 10 sectors (Technology, Consumer Cyclical, Financial Services, Healthcare, Energy, Communication Services, Industrials, Consumer Defensive, Real Estate, Utilities)
|
|
- 46 competitor relationships (direct_rival, same_sector, overlapping_products, supply_chain_adjacent)
|
|
- Seed script: `python -m services.symbol_registry.seed`
|
|
|
|
## Local Dev Environment
|
|
- Ubuntu dev machine, Python 3.12
|
|
- Virtual environment at `.venv/` — always use it for Python commands
|
|
- Node.js 24 via nvm — always load nvm before running Node/npm commands:
|
|
`export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" && nvm use 24`
|
|
- For tools not in `.venv/` (like `ruff`, `gh`), install via pip or apt as needed
|
|
- Docker available locally for image builds (but let CI handle pushes)
|
|
|
|
## Live Endpoints
|
|
- Dashboard: `https://stonks.celestium.life`
|
|
- Query API: `https://stonks-api.celestium.life`
|
|
- Symbol Registry: `https://stonks-registry.celestium.life`
|
|
- Trading Engine: `https://stonks-trading.celestium.life`
|
|
- Superset: `https://stonks-dash.celestium.life`
|
|
- Trino: `https://stonks-trino.celestium.life`
|
|
- Gitea: `https://git.celestium.life`
|
|
- Harbor Registry: `https://registry.celestium.life`
|
|
|
|
## Infrastructure
|
|
- Kubernetes cluster: 4x NixOS nodes (gremlin-1 through gremlin-4), reachable via `kubectl`, `virtctl`, `ssh root@gremlin-{1,2,3,4}`
|
|
- NixOS configs stored at `/etc/nixos` on gremlin-1, git-pushed to other hosts
|
|
- Ingress: Traefik, domain `*.celestium.life`
|
|
- Cert-Manager: `ca-issuer` (local CA) for internal services
|
|
- Container registry: `registry.celestium.life/stonks-oracle`
|
|
|
|
## CI/CD
|
|
- Woodpecker CI pipelines in `.woodpecker/` — triggered by push to `main` on Gitea
|
|
- Push to Gitea: `git push gitea main` — this is the primary push target
|
|
- ArgoCD watches Gitea `main` and auto-syncs beta/paper/live stages
|
|
- Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
|
|
- Images tagged as `registry.celestium.life/stonks-oracle/<service>:<sha>` and `:latest`
|
|
- Dashboard image: `frontend/Dockerfile` (multi-stage: node:24 → nginx-unprivileged on port 8080)
|
|
- Superset image: `docker/Dockerfile.superset` (apache/superset + trino + psycopg2)
|
|
- Python service images: `docker/Dockerfile` with `SERVICE_CMD` build arg
|
|
- Let CI handle image builds and pushes — do NOT manually `docker build && docker push`
|
|
- **Do NOT push directly to GitHub** — GitHub (`origin`) is the promotion target after CI builds and tests pass
|
|
- Promotion to GitHub: `git push origin main` (only after Woodpecker CI succeeds)
|
|
|
|
## Deployment Scripts
|
|
- `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available)
|
|
- `~/sources/kube/stonks-oracle/runmelast.sh` — teardown: Helm uninstall, clean resources (preserves DB/MinIO/Redis)
|
|
- After CI builds, deploy with: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle`
|
|
- Restart a single service: `kubectl rollout restart deployment/<name> -n stonks-oracle`
|
|
|
|
## Database Nuke & Rebuild
|
|
When a full reset is needed:
|
|
1. `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1)
|
|
2. `kubectl exec -n postgresql-service postgresql-1 -c postgres -- psql -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'stonks' AND pid <> pg_backend_pid();"`
|
|
3. `kubectl exec -n postgresql-service postgresql-1 -c postgres -- psql -U postgres -c "DROP DATABASE IF EXISTS stonks;"`
|
|
4. Flush Redis: clear all `stonks:*` keys to reset dedup markers
|
|
5. `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1)
|
|
6. Run seed: `POSTGRES_HOST=postgresql-rw.postgresql-service.svc.cluster.local POSTGRES_PASSWORD='St0nks0racl3!' POSTGRES_USER=stonks POSTGRES_DB=stonks .venv/bin/python -m services.symbol_registry.seed`
|
|
|
|
## API Secrets
|
|
- Stored as files in repo root (gitignored): `polygon.io.key`, `alpaca.key`, `alpaca.secret`, `alpaca.url`
|
|
- GitHub token at `/run/secrets/github_token` (on gremlin-1 only)
|
|
- Injected into K8s secrets via `runmefirst.sh` Helm `--set` flags
|
|
|
|
## Existing Cluster Services (do NOT redeploy these)
|
|
- PostgreSQL: `postgresql-rw.postgresql-service.svc.cluster.local:5432`
|
|
- Redis: `redis-master.redis-service.svc.cluster.local:6379` (password: in Helm secrets)
|
|
- MinIO: `minio.minio-service.svc.cluster.local:80` (API)
|
|
- Ollama: `ollama.ollama-service.svc.cluster.local:11434` (cluster-internal), also at `http://10.1.1.12:2701` (external), GPU: 4070 Ti Super 16GB
|
|
|
|
## Database Migrations
|
|
- Located in `infra/migrations/001_*.sql` through `030_*.sql`
|
|
- Applied automatically by `runmefirst.sh` in sorted order
|
|
- Next migration number: **038**
|
|
- Key migrations:
|
|
- 016: Global news interpolation (global_events, macro_impact_records, exposure_profiles, trend_projections)
|
|
- 017: Competitive intelligence (competitor_relationships, competitive_signal_records)
|
|
- 024: Trend history time-series table
|
|
- 026: AI agents management (ai_agents, agent_performance_log)
|
|
- 027: Agent variants (agent_variants table for A/B testing)
|
|
- 035: Model validation (prediction_snapshots, prediction_outcomes, signal_evidence_links, model_metric_snapshots, v_prediction_performance, v_source_performance)
|
|
|
|
## Key Conventions
|
|
- All services use `services/shared/config.py` for configuration via env vars
|
|
- Redis queues defined in `services/shared/redis_keys.py`
|
|
- Pydantic schemas in `services/shared/schemas.py`
|
|
- Helm chart in `infra/helm/stonks-oracle/`, all in `stonks-oracle` namespace
|
|
- Lakehouse DDL in `lakehouse/schemas/`
|
|
- Frontend proxies: `/api/` → query-api:8000, `/registry/` → symbol-registry:8000, `/risk/` → risk:8000
|
|
- Network policies: default-deny with explicit allow rules per service
|
|
|
|
## Signal Layers
|
|
- **Layer 1 (Company)**: document_impact_records → WeightedSignal → trend_windows
|
|
- **Layer 2 (Macro)**: global_events → macro_impact_records → WeightedSignal (toggle: `macro_enabled` in risk_configs)
|
|
- **Layer 3 (Competitive)**: pattern_matcher → signal_propagation → WeightedSignal (toggle: `competitive_enabled` in risk_configs)
|
|
- All three layers merge into the aggregation engine via the same WeightedSignal abstraction
|
|
- Each layer has an independent runtime toggle in risk_configs (no restart needed)
|
|
- Pattern-only and macro-only trend shifts are forced to informational mode (suppression safety)
|