docs: update steering docs with competitive layer, 50 companies, deployment lessons
This commit is contained in:
Celes Renata
@@ -18,10 +18,12 @@
|
||||
|
||||
## Testing
|
||||
- Python: `pytest` with `pytest-asyncio` for async code, tests in `tests/`
|
||||
- Property-based tests: Hypothesis with `@settings(max_examples=100)`, files prefixed `test_pbt_*`
|
||||
- Frontend: Vitest + MSW (Mock Service Worker) for deterministic API mocking, tests in `frontend/src/test/`
|
||||
- Run Python tests: `python -m pytest tests/ -x --tb=short -q`
|
||||
- Run frontend tests: `cd frontend && npx vitest --run`
|
||||
- Lint Python: `nix-shell -p ruff --run "ruff check services/"`
|
||||
- Pre-existing test failures (not regressions): `test_extractor_prompts.py`, `test_extractor_schemas.py`, `test_filings_adapter.py`, `test_ollama_client.py`
|
||||
|
||||
## CI/CD — GitHub Actions
|
||||
- Workflow: `.github/workflows/build.yml`
|
||||
@@ -29,22 +31,44 @@
|
||||
- Jobs:
|
||||
- `lint-and-test`: ruff lint + pytest + frontend vitest (Node 24)
|
||||
- `build-services`: matrix build of all Python services → GHCR
|
||||
- `build-dashboard`: frontend/Dockerfile → GHCR
|
||||
- `build-dashboard`: frontend/Dockerfile → GHCR (TypeScript strict mode — catches unused imports)
|
||||
- `build-superset`: docker/Dockerfile.superset → GHCR
|
||||
- CI handles all image builds and pushes — do NOT manually docker push
|
||||
- Check CI: `nix-shell -p gh --run "gh run list -L 3"`
|
||||
- Check CI: `nix-shell -p gh --run "gh run list -L 3"` or `gh run list -L 3`
|
||||
- Re-run failed: `nix-shell -p gh --run "gh run rerun <id> --failed"`
|
||||
- View failure logs: `gh run view <id> --log-failed`
|
||||
|
||||
## Deploy
|
||||
- Full deploy/redeploy: `~/sources/kube/stonks-oracle/runmefirst.sh`
|
||||
- Full teardown: `~/sources/kube/stonks-oracle/runmelast.sh`
|
||||
- Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1)
|
||||
- Full teardown: `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1)
|
||||
- Quick Helm upgrade: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle`
|
||||
- Restart single service: `kubectl rollout restart deployment/<name> -n stonks-oracle`
|
||||
- Restart multiple: `kubectl rollout restart deployment/aggregation deployment/query-api deployment/dashboard -n stonks-oracle`
|
||||
- Check pods: `kubectl get pods -n stonks-oracle`
|
||||
- Check logs: `kubectl logs deployment/<name> -n stonks-oracle --tail=30`
|
||||
|
||||
## Manually Triggering Ingestion
|
||||
The scheduler runs on a cadence, but to trigger immediately:
|
||||
```python
|
||||
# From a scheduler pod:
|
||||
kubectl exec -n stonks-oracle <scheduler-pod> -- python -c "
|
||||
import redis, json, asyncio, asyncpg
|
||||
async def enqueue():
|
||||
pool = await asyncpg.create_pool(dsn='postgresql://stonks:<password>@postgresql-rw.postgresql-service.svc.cluster.local:5432/stonks')
|
||||
r = redis.from_url('redis://:<password>@redis-master.redis-service.svc.cluster.local:6379/0')
|
||||
rows = await pool.fetch('SELECT s.id AS source_id, s.source_type, s.config, c.id AS company_id, c.ticker FROM sources s JOIN companies c ON c.id = s.company_id WHERE s.active = TRUE AND c.active = TRUE')
|
||||
for row in rows:
|
||||
cfg = row['config'] if isinstance(row['config'], dict) else {}
|
||||
r.rpush('stonks:queue:ingestion', json.dumps({'source_id': str(row['source_id']), 'source_type': row['source_type'], 'ticker': row['ticker'], 'company_id': str(row['company_id']), 'config': cfg}))
|
||||
await pool.close()
|
||||
asyncio.run(enqueue())
|
||||
"
|
||||
```
|
||||
Ingestion jobs MUST include `source_id`, `source_type`, `ticker`, `company_id`, and `config`.
|
||||
|
||||
## Git Conventions
|
||||
- Commit after each completed phase task
|
||||
- Commit message format: `phase N: short description`
|
||||
- Commit message format: `feat:`, `fix:`, `phase N:` prefix
|
||||
- Push to `main` triggers CI
|
||||
|
||||
## Code Style
|
||||
@@ -53,8 +77,17 @@
|
||||
- FastAPI for HTTP services
|
||||
- asyncio + asyncpg/aioredis for async I/O
|
||||
- Minimal dependencies, prefer stdlib where possible
|
||||
- Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query
|
||||
- Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query, Recharts for charts
|
||||
- UUID fields from asyncpg must be converted to str via `_row_dict()` helpers
|
||||
- asyncpg interval parameters must be Python `timedelta` objects, not SQL strings
|
||||
- Recharts v3 callbacks: do NOT add explicit type annotations on `formatter`/`tickFormatter` — let TypeScript infer
|
||||
|
||||
## Common Pitfalls
|
||||
- asyncpg expects `timedelta` for `$N::interval` params, not strings like `'7 days'`
|
||||
- asyncpg expects UUID objects/strings for `$N::uuid` params — synthetic IDs like `pattern:AAPL:earnings:7d` will fail
|
||||
- The `competitor_relationships` table uses UUID company IDs — queries must join through `companies` to match by ticker
|
||||
- The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI
|
||||
- Ingestion jobs require `source_id` from the `sources` table — don't just pass `ticker`
|
||||
|
||||
## Documentation
|
||||
- Do NOT create large summary/success markdown files after each step
|
||||
|
||||
Reference in New Issue
Block a user