docs: update steering docs with competitive layer, 50 companies, deployment lessons

.kiro/steering/development-process.md

@@ -18,10 +18,12 @@
 
 ## Testing
 - Python: `pytest` with `pytest-asyncio` for async code, tests in `tests/`
+- Property-based tests: Hypothesis with `@settings(max_examples=100)`, files prefixed `test_pbt_*`
 - Frontend: Vitest + MSW (Mock Service Worker) for deterministic API mocking, tests in `frontend/src/test/`
 - Run Python tests: `python -m pytest tests/ -x --tb=short -q`
 - Run frontend tests: `cd frontend && npx vitest --run`
 - Lint Python: `nix-shell -p ruff --run "ruff check services/"`
+- Pre-existing test failures (not regressions): `test_extractor_prompts.py`, `test_extractor_schemas.py`, `test_filings_adapter.py`, `test_ollama_client.py`
 
 ## CI/CD — GitHub Actions
 - Workflow: `.github/workflows/build.yml`
@@ -29,22 +31,44 @@
 - Jobs:
   - `lint-and-test`: ruff lint + pytest + frontend vitest (Node 24)
   - `build-services`: matrix build of all Python services → GHCR
-  - `build-dashboard`: frontend/Dockerfile → GHCR
+  - `build-dashboard`: frontend/Dockerfile → GHCR (TypeScript strict mode — catches unused imports)
   - `build-superset`: docker/Dockerfile.superset → GHCR
 - CI handles all image builds and pushes — do NOT manually docker push
-- Check CI: `nix-shell -p gh --run "gh run list -L 3"`
+- Check CI: `nix-shell -p gh --run "gh run list -L 3"` or `gh run list -L 3`
 - Re-run failed: `nix-shell -p gh --run "gh run rerun <id> --failed"`
+- View failure logs: `gh run view <id> --log-failed`
 
 ## Deploy
-- Full deploy/redeploy: `~/sources/kube/stonks-oracle/runmefirst.sh`
-- Full teardown: `~/sources/kube/stonks-oracle/runmelast.sh`
+- Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1)
+- Full teardown: `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1)
 - Quick Helm upgrade: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle`
 - Restart single service: `kubectl rollout restart deployment/<name> -n stonks-oracle`
+- Restart multiple: `kubectl rollout restart deployment/aggregation deployment/query-api deployment/dashboard -n stonks-oracle`
 - Check pods: `kubectl get pods -n stonks-oracle`
+- Check logs: `kubectl logs deployment/<name> -n stonks-oracle --tail=30`
+
+## Manually Triggering Ingestion
+The scheduler runs on a cadence, but to trigger immediately:
+```python
+# From a scheduler pod:
+kubectl exec -n stonks-oracle <scheduler-pod> -- python -c "
+import redis, json, asyncio, asyncpg
+async def enqueue():
+    pool = await asyncpg.create_pool(dsn='postgresql://stonks:<password>@postgresql-rw.postgresql-service.svc.cluster.local:5432/stonks')
+    r = redis.from_url('redis://:<password>@redis-master.redis-service.svc.cluster.local:6379/0')
+    rows = await pool.fetch('SELECT s.id AS source_id, s.source_type, s.config, c.id AS company_id, c.ticker FROM sources s JOIN companies c ON c.id = s.company_id WHERE s.active = TRUE AND c.active = TRUE')
+    for row in rows:
+        cfg = row['config'] if isinstance(row['config'], dict) else {}
+        r.rpush('stonks:queue:ingestion', json.dumps({'source_id': str(row['source_id']), 'source_type': row['source_type'], 'ticker': row['ticker'], 'company_id': str(row['company_id']), 'config': cfg}))
+    await pool.close()
+asyncio.run(enqueue())
+"
+```
+Ingestion jobs MUST include `source_id`, `source_type`, `ticker`, `company_id`, and `config`.
 
 ## Git Conventions
 - Commit after each completed phase task
-- Commit message format: `phase N: short description`
+- Commit message format: `feat:`, `fix:`, `phase N:` prefix
 - Push to `main` triggers CI
 
 ## Code Style
@@ -53,8 +77,17 @@
 - FastAPI for HTTP services
 - asyncio + asyncpg/aioredis for async I/O
 - Minimal dependencies, prefer stdlib where possible
-- Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query
+- Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query, Recharts for charts
 - UUID fields from asyncpg must be converted to str via `_row_dict()` helpers
+- asyncpg interval parameters must be Python `timedelta` objects, not SQL strings
+- Recharts v3 callbacks: do NOT add explicit type annotations on `formatter`/`tickFormatter` — let TypeScript infer
+
+## Common Pitfalls
+- asyncpg expects `timedelta` for `$N::interval` params, not strings like `'7 days'`
+- asyncpg expects UUID objects/strings for `$N::uuid` params — synthetic IDs like `pattern:AAPL:earnings:7d` will fail
+- The `competitor_relationships` table uses UUID company IDs — queries must join through `companies` to match by ticker
+- The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI
+- Ingestion jobs require `source_id` from the `sources` table — don't just pass `ticker`
 
 ## Documentation
 - Do NOT create large summary/success markdown files after each step

.kiro/steering/project-context.md

@@ -4,6 +4,16 @@
 Stonks Oracle is a Kubernetes-native AI market intelligence and paper-trading platform.
 Python monorepo with services under `services/`, infrastructure under `infra/`, lakehouse schemas under `lakehouse/`, frontend React dashboard under `frontend/`, and dashboards under `dashboards/`.
 
+Three-layer signal aggregation engine:
+1. **Company-specific signals** — document intelligence from news, filings, market data
+2. **Macro signals** — global news interpolation, geopolitical event classification, exposure-based impact scoring
+3. **Competitive signals** — historical pattern mining, cross-company signal propagation, competitor relationship management
+
+## Tracked Universe
+- 50 companies across 10 sectors (Technology, Consumer Cyclical, Financial Services, Healthcare, Energy, Communication Services, Industrials, Consumer Defensive, Real Estate, Utilities)
+- 46 competitor relationships (direct_rival, same_sector, overlapping_products, supply_chain_adjacent)
+- Seed script: `python -m services.symbol_registry.seed`
+
 ## Local Dev Environment
 - NixOS dev environment, Python 3.12
 - Virtual environment at `.venv/` — always use it for Python commands
@@ -36,22 +46,39 @@ Python monorepo with services under `services/`, infrastructure under `infra/`,
 - Check CI status: `nix-shell -p gh --run "gh run list -L 3"`
 
 ## Deployment Scripts
-- `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart
+- `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available)
 - `~/sources/kube/stonks-oracle/runmelast.sh` — teardown: Helm uninstall, clean resources (preserves DB/MinIO/Redis)
 - After CI builds, deploy with: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle`
 - Restart a single service: `kubectl rollout restart deployment/<name> -n stonks-oracle`
 
+## Database Nuke & Rebuild
+When a full reset is needed:
+1. `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1)
+2. `kubectl exec -n postgresql-service postgresql-1 -c postgres -- psql -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'stonks' AND pid <> pg_backend_pid();"`
+3. `kubectl exec -n postgresql-service postgresql-1 -c postgres -- psql -U postgres -c "DROP DATABASE IF EXISTS stonks;"`
+4. Flush Redis: clear all `stonks:*` keys to reset dedup markers
+5. `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1)
+6. Run seed: `POSTGRES_HOST=postgresql-rw.postgresql-service.svc.cluster.local POSTGRES_PASSWORD='St0nks0racl3!' POSTGRES_USER=stonks POSTGRES_DB=stonks .venv/bin/python -m services.symbol_registry.seed`
+
 ## API Secrets
 - Stored as files in repo root (gitignored): `polygon.io.key`, `alpaca.key`, `alpaca.secret`, `alpaca.url`
-- GitHub token at `/run/secrets/github_token`
+- GitHub token at `/run/secrets/github_token` (on gremlin-1 only)
 - Injected into K8s secrets via `runmefirst.sh` Helm `--set` flags
 
 ## Existing Cluster Services (do NOT redeploy these)
 - PostgreSQL: `postgresql-rw.postgresql-service.svc.cluster.local:5432`
-- Redis: `redis-master.redis-service.svc.cluster.local:6379`
+- Redis: `redis-master.redis-service.svc.cluster.local:6379` (password: in Helm secrets)
 - MinIO: `minio.minio-service.svc.cluster.local:80` (API)
 - Ollama: `ollama.ollama-service.svc.cluster.local:11434` (cluster-internal), also at `http://10.1.1.12:2701` (external), GPU: 4070 Ti Super 16GB
 
+## Database Migrations
+- Located in `infra/migrations/001_*.sql` through `017_*.sql`
+- Applied automatically by `runmefirst.sh` in sorted order
+- Next migration number: **018**
+- Key migrations:
+  - 016: Global news interpolation (global_events, macro_impact_records, exposure_profiles, trend_projections)
+  - 017: Competitive intelligence (competitor_relationships, competitive_signal_records)
+
 ## Key Conventions
 - All services use `services/shared/config.py` for configuration via env vars
 - Redis queues defined in `services/shared/redis_keys.py`
@@ -60,3 +87,11 @@ Python monorepo with services under `services/`, infrastructure under `infra/`,
 - Lakehouse DDL in `lakehouse/schemas/`
 - Frontend proxies: `/api/` → query-api:8000, `/registry/` → symbol-registry:8000, `/risk/` → risk:8000
 - Network policies: default-deny with explicit allow rules per service
+
+## Signal Layers
+- **Layer 1 (Company)**: document_impact_records → WeightedSignal → trend_windows
+- **Layer 2 (Macro)**: global_events → macro_impact_records → WeightedSignal (toggle: `macro_enabled` in risk_configs)
+- **Layer 3 (Competitive)**: pattern_matcher → signal_propagation → WeightedSignal (toggle: `competitive_enabled` in risk_configs)
+- All three layers merge into the aggregation engine via the same WeightedSignal abstraction
+- Each layer has an independent runtime toggle in risk_configs (no restart needed)
+- Pattern-only and macro-only trend shifts are forced to informational mode (suppression safety)