diff --git a/.kiro/steering/development-process.md b/.kiro/steering/development-process.md index 166dd3b..394d048 100644 --- a/.kiro/steering/development-process.md +++ b/.kiro/steering/development-process.md @@ -18,10 +18,12 @@ ## Testing - Python: `pytest` with `pytest-asyncio` for async code, tests in `tests/` +- Property-based tests: Hypothesis with `@settings(max_examples=100)`, files prefixed `test_pbt_*` - Frontend: Vitest + MSW (Mock Service Worker) for deterministic API mocking, tests in `frontend/src/test/` - Run Python tests: `python -m pytest tests/ -x --tb=short -q` - Run frontend tests: `cd frontend && npx vitest --run` - Lint Python: `nix-shell -p ruff --run "ruff check services/"` +- Pre-existing test failures (not regressions): `test_extractor_prompts.py`, `test_extractor_schemas.py`, `test_filings_adapter.py`, `test_ollama_client.py` ## CI/CD — GitHub Actions - Workflow: `.github/workflows/build.yml` @@ -29,22 +31,44 @@ - Jobs: - `lint-and-test`: ruff lint + pytest + frontend vitest (Node 24) - `build-services`: matrix build of all Python services → GHCR - - `build-dashboard`: frontend/Dockerfile → GHCR + - `build-dashboard`: frontend/Dockerfile → GHCR (TypeScript strict mode — catches unused imports) - `build-superset`: docker/Dockerfile.superset → GHCR - CI handles all image builds and pushes — do NOT manually docker push -- Check CI: `nix-shell -p gh --run "gh run list -L 3"` +- Check CI: `nix-shell -p gh --run "gh run list -L 3"` or `gh run list -L 3` - Re-run failed: `nix-shell -p gh --run "gh run rerun --failed"` +- View failure logs: `gh run view --log-failed` ## Deploy -- Full deploy/redeploy: `~/sources/kube/stonks-oracle/runmefirst.sh` -- Full teardown: `~/sources/kube/stonks-oracle/runmelast.sh` +- Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1) +- Full teardown: `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1) - Quick Helm upgrade: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle` - Restart single service: `kubectl rollout restart deployment/ -n stonks-oracle` +- Restart multiple: `kubectl rollout restart deployment/aggregation deployment/query-api deployment/dashboard -n stonks-oracle` - Check pods: `kubectl get pods -n stonks-oracle` +- Check logs: `kubectl logs deployment/ -n stonks-oracle --tail=30` + +## Manually Triggering Ingestion +The scheduler runs on a cadence, but to trigger immediately: +```python +# From a scheduler pod: +kubectl exec -n stonks-oracle -- python -c " +import redis, json, asyncio, asyncpg +async def enqueue(): + pool = await asyncpg.create_pool(dsn='postgresql://stonks:@postgresql-rw.postgresql-service.svc.cluster.local:5432/stonks') + r = redis.from_url('redis://:@redis-master.redis-service.svc.cluster.local:6379/0') + rows = await pool.fetch('SELECT s.id AS source_id, s.source_type, s.config, c.id AS company_id, c.ticker FROM sources s JOIN companies c ON c.id = s.company_id WHERE s.active = TRUE AND c.active = TRUE') + for row in rows: + cfg = row['config'] if isinstance(row['config'], dict) else {} + r.rpush('stonks:queue:ingestion', json.dumps({'source_id': str(row['source_id']), 'source_type': row['source_type'], 'ticker': row['ticker'], 'company_id': str(row['company_id']), 'config': cfg})) + await pool.close() +asyncio.run(enqueue()) +" +``` +Ingestion jobs MUST include `source_id`, `source_type`, `ticker`, `company_id`, and `config`. ## Git Conventions - Commit after each completed phase task -- Commit message format: `phase N: short description` +- Commit message format: `feat:`, `fix:`, `phase N:` prefix - Push to `main` triggers CI ## Code Style @@ -53,8 +77,17 @@ - FastAPI for HTTP services - asyncio + asyncpg/aioredis for async I/O - Minimal dependencies, prefer stdlib where possible -- Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query +- Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query, Recharts for charts - UUID fields from asyncpg must be converted to str via `_row_dict()` helpers +- asyncpg interval parameters must be Python `timedelta` objects, not SQL strings +- Recharts v3 callbacks: do NOT add explicit type annotations on `formatter`/`tickFormatter` — let TypeScript infer + +## Common Pitfalls +- asyncpg expects `timedelta` for `$N::interval` params, not strings like `'7 days'` +- asyncpg expects UUID objects/strings for `$N::uuid` params — synthetic IDs like `pattern:AAPL:earnings:7d` will fail +- The `competitor_relationships` table uses UUID company IDs — queries must join through `companies` to match by ticker +- The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI +- Ingestion jobs require `source_id` from the `sources` table — don't just pass `ticker` ## Documentation - Do NOT create large summary/success markdown files after each step diff --git a/.kiro/steering/project-context.md b/.kiro/steering/project-context.md index cab269c..b55c7fc 100644 --- a/.kiro/steering/project-context.md +++ b/.kiro/steering/project-context.md @@ -4,6 +4,16 @@ Stonks Oracle is a Kubernetes-native AI market intelligence and paper-trading platform. Python monorepo with services under `services/`, infrastructure under `infra/`, lakehouse schemas under `lakehouse/`, frontend React dashboard under `frontend/`, and dashboards under `dashboards/`. +Three-layer signal aggregation engine: +1. **Company-specific signals** — document intelligence from news, filings, market data +2. **Macro signals** — global news interpolation, geopolitical event classification, exposure-based impact scoring +3. **Competitive signals** — historical pattern mining, cross-company signal propagation, competitor relationship management + +## Tracked Universe +- 50 companies across 10 sectors (Technology, Consumer Cyclical, Financial Services, Healthcare, Energy, Communication Services, Industrials, Consumer Defensive, Real Estate, Utilities) +- 46 competitor relationships (direct_rival, same_sector, overlapping_products, supply_chain_adjacent) +- Seed script: `python -m services.symbol_registry.seed` + ## Local Dev Environment - NixOS dev environment, Python 3.12 - Virtual environment at `.venv/` — always use it for Python commands @@ -36,22 +46,39 @@ Python monorepo with services under `services/`, infrastructure under `infra/`, - Check CI status: `nix-shell -p gh --run "gh run list -L 3"` ## Deployment Scripts -- `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart +- `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available) - `~/sources/kube/stonks-oracle/runmelast.sh` — teardown: Helm uninstall, clean resources (preserves DB/MinIO/Redis) - After CI builds, deploy with: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle` - Restart a single service: `kubectl rollout restart deployment/ -n stonks-oracle` +## Database Nuke & Rebuild +When a full reset is needed: +1. `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1) +2. `kubectl exec -n postgresql-service postgresql-1 -c postgres -- psql -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'stonks' AND pid <> pg_backend_pid();"` +3. `kubectl exec -n postgresql-service postgresql-1 -c postgres -- psql -U postgres -c "DROP DATABASE IF EXISTS stonks;"` +4. Flush Redis: clear all `stonks:*` keys to reset dedup markers +5. `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1) +6. Run seed: `POSTGRES_HOST=postgresql-rw.postgresql-service.svc.cluster.local POSTGRES_PASSWORD='St0nks0racl3!' POSTGRES_USER=stonks POSTGRES_DB=stonks .venv/bin/python -m services.symbol_registry.seed` + ## API Secrets - Stored as files in repo root (gitignored): `polygon.io.key`, `alpaca.key`, `alpaca.secret`, `alpaca.url` -- GitHub token at `/run/secrets/github_token` +- GitHub token at `/run/secrets/github_token` (on gremlin-1 only) - Injected into K8s secrets via `runmefirst.sh` Helm `--set` flags ## Existing Cluster Services (do NOT redeploy these) - PostgreSQL: `postgresql-rw.postgresql-service.svc.cluster.local:5432` -- Redis: `redis-master.redis-service.svc.cluster.local:6379` +- Redis: `redis-master.redis-service.svc.cluster.local:6379` (password: in Helm secrets) - MinIO: `minio.minio-service.svc.cluster.local:80` (API) - Ollama: `ollama.ollama-service.svc.cluster.local:11434` (cluster-internal), also at `http://10.1.1.12:2701` (external), GPU: 4070 Ti Super 16GB +## Database Migrations +- Located in `infra/migrations/001_*.sql` through `017_*.sql` +- Applied automatically by `runmefirst.sh` in sorted order +- Next migration number: **018** +- Key migrations: + - 016: Global news interpolation (global_events, macro_impact_records, exposure_profiles, trend_projections) + - 017: Competitive intelligence (competitor_relationships, competitive_signal_records) + ## Key Conventions - All services use `services/shared/config.py` for configuration via env vars - Redis queues defined in `services/shared/redis_keys.py` @@ -60,3 +87,11 @@ Python monorepo with services under `services/`, infrastructure under `infra/`, - Lakehouse DDL in `lakehouse/schemas/` - Frontend proxies: `/api/` → query-api:8000, `/registry/` → symbol-registry:8000, `/risk/` → risk:8000 - Network policies: default-deny with explicit allow rules per service + +## Signal Layers +- **Layer 1 (Company)**: document_impact_records → WeightedSignal → trend_windows +- **Layer 2 (Macro)**: global_events → macro_impact_records → WeightedSignal (toggle: `macro_enabled` in risk_configs) +- **Layer 3 (Competitive)**: pattern_matcher → signal_propagation → WeightedSignal (toggle: `competitive_enabled` in risk_configs) +- All three layers merge into the aggregation engine via the same WeightedSignal abstraction +- Each layer has an independent runtime toggle in risk_configs (no restart needed) +- Pattern-only and macro-only trend shifts are forced to informational mode (suppression safety)