admin/stonks-oracle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
376fcb4bb4fa3d53a2ba1bfe92f87ae38a55e6b6
stonks-oracle/.kiro/steering/project-context.md
T
Celes Renata 376fcb4bb4
ci/woodpecker/push/test Pipeline was successful
Details
ci/woodpecker/push/build-2 Pipeline was successful
Details
ci/woodpecker/push/build-1 Pipeline was successful
Details
ci/woodpecker/push/build-3 Pipeline was successful
Details
ci/woodpecker/push/finalize Pipeline was successful
Details
Build and Push / lint-and-test (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Details
Build and Push / build-dashboard (push) Has been cancelled
Details
Build and Push / build-superset (push) Has been cancelled
Details
Build and Push / integration-test (push) Has been cancelled
Details
Build and Push / beta-gate (push) Has been cancelled
Details
feat: add 12 integrity check saved queries for SQL Explorer (migration 037) 
Prefixed with ⚕ for easy identification. Checks: duplicate snapshots,
orphaned links, evidence count mismatches, contribution score sums,
canonical key consistency, out-of-range values, unmatched snapshots,
zero evidence rate, duplicate count mismatches, missing prices,
outcome integrity, and pipeline health summary.
2026-05-01 19:20:56 +00:00

7.0 KiB
Raw Blame History

Stonks Oracle — Project Context

Overview

Stonks Oracle is a Kubernetes-native AI market intelligence and paper-trading platform. Python monorepo with services under services/, infrastructure under infra/, lakehouse schemas under lakehouse/, frontend React dashboard under frontend/, and dashboards under dashboards/.

Three-layer signal aggregation engine:

  1. Company-specific signals — document intelligence from news, filings, market data
  2. Macro signals — global news interpolation, geopolitical event classification, exposure-based impact scoring
  3. Competitive signals — historical pattern mining, cross-company signal propagation, competitor relationship management

Tracked Universe

  • 50 companies across 10 sectors (Technology, Consumer Cyclical, Financial Services, Healthcare, Energy, Communication Services, Industrials, Consumer Defensive, Real Estate, Utilities)
  • 46 competitor relationships (direct_rival, same_sector, overlapping_products, supply_chain_adjacent)
  • Seed script: python -m services.symbol_registry.seed

Local Dev Environment

  • Ubuntu dev machine, Python 3.12
  • Virtual environment at .venv/ — always use it for Python commands
  • Node.js 24 via nvm — always load nvm before running Node/npm commands: export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" && nvm use 24
  • For tools not in .venv/ (like ruff, gh), install via pip or apt as needed
  • Docker available locally for image builds (but let CI handle pushes)

Live Endpoints

  • Dashboard: https://stonks.celestium.life
  • Query API: https://stonks-api.celestium.life
  • Symbol Registry: https://stonks-registry.celestium.life
  • Trading Engine: https://stonks-trading.celestium.life
  • Superset: https://stonks-dash.celestium.life
  • Trino: https://stonks-trino.celestium.life
  • Gitea: https://git.celestium.life
  • Harbor Registry: https://registry.celestium.life

Infrastructure

  • Kubernetes cluster: 4x NixOS nodes (gremlin-1 through gremlin-4), reachable via kubectl, virtctl, ssh root@gremlin-{1,2,3,4}
  • NixOS configs stored at /etc/nixos on gremlin-1, git-pushed to other hosts
  • Ingress: Traefik, domain *.celestium.life
  • Cert-Manager: ca-issuer (local CA) for internal services
  • Container registry: registry.celestium.life/stonks-oracle

CI/CD

  • Woodpecker CI pipelines in .woodpecker/ — triggered by push to main on Gitea
  • Push to Gitea: git push gitea main — this is the primary push target
  • ArgoCD watches Gitea main and auto-syncs beta/paper/live stages
  • Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
  • Images tagged as registry.celestium.life/stonks-oracle/<service>:<sha> and :latest
  • Dashboard image: frontend/Dockerfile (multi-stage: node:24 → nginx-unprivileged on port 8080)
  • Superset image: docker/Dockerfile.superset (apache/superset + trino + psycopg2)
  • Python service images: docker/Dockerfile with SERVICE_CMD build arg
  • Let CI handle image builds and pushes — do NOT manually docker build && docker push
  • Do NOT push directly to GitHub — GitHub (origin) is the promotion target after CI builds and tests pass
  • Promotion to GitHub: git push origin main (only after Woodpecker CI succeeds)

Deployment Scripts

  • ~/sources/kube/stonks-oracle/runmefirst.sh — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available)
  • ~/sources/kube/stonks-oracle/runmelast.sh — teardown: Helm uninstall, clean resources (preserves DB/MinIO/Redis)
  • After CI builds, deploy with: helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle
  • Restart a single service: kubectl rollout restart deployment/<name> -n stonks-oracle

Database Nuke & Rebuild

When a full reset is needed:

  1. bash ~/sources/kube/stonks-oracle/runmelast.sh (from gremlin-1)
  2. kubectl exec -n postgresql-service postgresql-1 -c postgres -- psql -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'stonks' AND pid <> pg_backend_pid();"
  3. kubectl exec -n postgresql-service postgresql-1 -c postgres -- psql -U postgres -c "DROP DATABASE IF EXISTS stonks;"
  4. Flush Redis: clear all stonks:* keys to reset dedup markers
  5. bash ~/sources/kube/stonks-oracle/runmefirst.sh (from gremlin-1)
  6. Run seed: POSTGRES_HOST=postgresql-rw.postgresql-service.svc.cluster.local POSTGRES_PASSWORD='St0nks0racl3!' POSTGRES_USER=stonks POSTGRES_DB=stonks .venv/bin/python -m services.symbol_registry.seed

API Secrets

  • Stored as files in repo root (gitignored): polygon.io.key, alpaca.key, alpaca.secret, alpaca.url
  • GitHub token at /run/secrets/github_token (on gremlin-1 only)
  • Injected into K8s secrets via runmefirst.sh Helm --set flags

Existing Cluster Services (do NOT redeploy these)

  • PostgreSQL: postgresql-rw.postgresql-service.svc.cluster.local:5432
  • Redis: redis-master.redis-service.svc.cluster.local:6379 (password: in Helm secrets)
  • MinIO: minio.minio-service.svc.cluster.local:80 (API)
  • Ollama: ollama.ollama-service.svc.cluster.local:11434 (cluster-internal), also at http://10.1.1.12:2701 (external), GPU: 4070 Ti Super 16GB

Database Migrations

  • Located in infra/migrations/001_*.sql through 030_*.sql
  • Applied automatically by runmefirst.sh in sorted order
  • Next migration number: 038
  • Key migrations:
    • 016: Global news interpolation (global_events, macro_impact_records, exposure_profiles, trend_projections)
    • 017: Competitive intelligence (competitor_relationships, competitive_signal_records)
    • 024: Trend history time-series table
    • 026: AI agents management (ai_agents, agent_performance_log)
    • 027: Agent variants (agent_variants table for A/B testing)
    • 035: Model validation (prediction_snapshots, prediction_outcomes, signal_evidence_links, model_metric_snapshots, v_prediction_performance, v_source_performance)

Key Conventions

  • All services use services/shared/config.py for configuration via env vars
  • Redis queues defined in services/shared/redis_keys.py
  • Pydantic schemas in services/shared/schemas.py
  • Helm chart in infra/helm/stonks-oracle/, all in stonks-oracle namespace
  • Lakehouse DDL in lakehouse/schemas/
  • Frontend proxies: /api/ → query-api:8000, /registry/ → symbol-registry:8000, /risk/ → risk:8000
  • Network policies: default-deny with explicit allow rules per service

Signal Layers

  • Layer 1 (Company): document_impact_records → WeightedSignal → trend_windows
  • Layer 2 (Macro): global_events → macro_impact_records → WeightedSignal (toggle: macro_enabled in risk_configs)
  • Layer 3 (Competitive): pattern_matcher → signal_propagation → WeightedSignal (toggle: competitive_enabled in risk_configs)
  • All three layers merge into the aggregation engine via the same WeightedSignal abstraction
  • Each layer has an independent runtime toggle in risk_configs (no restart needed)
  • Pattern-only and macro-only trend shifts are forced to informational mode (suppression safety)
Reference in New Issue View Git Blame Copy Permalink