stonks-oracle/.kiro/steering/development-process.md

# Development Process — Test-Develop-Debug

## Local Environment
- Ubuntu dev machine, Python 3.12, virtualenv at `.venv/`
- Always use `.venv/bin/python` or activate with `source .venv/bin/activate` before running Python commands
- Node.js 24 via nvm — always load nvm before running Node/npm/npx commands:
  `export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" && nvm use 24`
- For tools not available in `.venv/` (ruff, gh, etc.), install via pip or apt as needed
- Frontend tests: load nvm first, then `cd frontend && npx vitest --run`
- Python tests: `.venv/bin/ruff check services/` then `.venv/bin/python -m pytest tests/ -x --tb=short -q`

## Workflow
1. Write or update tests for the target behavior
2. Implement the minimal code to pass
3. Debug failures, fix, re-run
4. Commit and push — CI builds images automatically
5. Deploy: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle`
6. Restart changed services: `kubectl rollout restart deployment/<name> -n stonks-oracle`

## Testing
- Python: `pytest` with `pytest-asyncio` for async code, tests in `tests/`
- Property-based tests: Hypothesis with `@settings(max_examples=100)`, files prefixed `test_pbt_*`
- Frontend: Vitest + MSW (Mock Service Worker) for deterministic API mocking, tests in `frontend/src/test/`
- Run Python tests: `python -m pytest tests/ -x --tb=short -q`
- Run frontend tests: `cd frontend && npx vitest --run`
- Lint Python: `.venv/bin/ruff check services/`
- **Always run `.venv/bin/ruff check services/` before committing Python changes** — CI will reject the push if ruff fails
- Ruff auto-fix: `.venv/bin/ruff check --fix services/` (fixes import sorting and other auto-fixable issues)
- Ruff is pinned to `ruff==0.15.10` in `requirements.txt` — CI uses the same version
- Ruff config: `ruff.toml` with `known-first-party = ["services"]` for consistent import sorting
- Pre-existing test failures (not regressions): `test_extractor_prompts.py`, `test_extractor_schemas.py`, `test_filings_adapter.py`, `test_ollama_client.py`

## CI/CD — Woodpecker CI (Gitea) → GitHub promotion
- Woodpecker pipelines in `.woodpecker/` — triggered by push to `main` on Gitea
- Push to Gitea: `git push gitea main`
- Gitea remote: `http://admin:<password>@10.1.1.12:30300/admin/stonks-oracle.git`
- Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
- ArgoCD watches Gitea `main` and auto-syncs beta/paper/live stages
- **Do NOT push directly to GitHub** — GitHub is the promotion target after CI passes
- Once Woodpecker builds and tests pass, code is promoted to GitHub (`git push origin main`)
- CI handles all image builds and pushes — do NOT manually docker push
- Check Woodpecker CI status from the Gitea web UI or Woodpecker dashboard

## Deploy
- Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1)
- Full teardown: `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1)
- Quick Helm upgrade: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle`
- Restart single service: `kubectl rollout restart deployment/<name> -n stonks-oracle`
- Restart multiple: `kubectl rollout restart deployment/aggregation deployment/query-api deployment/dashboard -n stonks-oracle`
- Check pods: `kubectl get pods -n stonks-oracle`
- Check logs: `kubectl logs deployment/<name> -n stonks-oracle --tail=30`

## Manually Triggering Ingestion
The scheduler runs on a cadence, but to trigger immediately:
```python
# From a scheduler pod:
kubectl exec -n stonks-oracle <scheduler-pod> -- python -c "
import redis, json, asyncio, asyncpg
async def enqueue():
    pool = await asyncpg.create_pool(dsn='postgresql://stonks:<password>@postgresql-rw.postgresql-service.svc.cluster.local:5432/stonks')
    r = redis.from_url('redis://:<password>@redis-master.redis-service.svc.cluster.local:6379/0')
    rows = await pool.fetch('SELECT s.id AS source_id, s.source_type, s.config, c.id AS company_id, c.ticker FROM sources s JOIN companies c ON c.id = s.company_id WHERE s.active = TRUE AND c.active = TRUE')
    for row in rows:
        cfg = row['config'] if isinstance(row['config'], dict) else {}
        r.rpush('stonks:queue:ingestion', json.dumps({'source_id': str(row['source_id']), 'source_type': row['source_type'], 'ticker': row['ticker'], 'company_id': str(row['company_id']), 'config': cfg}))
    await pool.close()
asyncio.run(enqueue())
"
```
Ingestion jobs MUST include `source_id`, `source_type`, `ticker`, `company_id`, and `config`.

## Git Conventions
- Commit after each completed phase task
- Commit message format: `feat:`, `fix:`, `phase N:` prefix
- Always push to Gitea: `git push gitea main`
- Do NOT push to GitHub (`origin`) directly — GitHub is the promotion target after CI passes
- ArgoCD syncs from Gitea automatically

## Code Style
- Python 3.12, type hints everywhere
- Pydantic for data validation
- FastAPI for HTTP services
- asyncio + asyncpg/aioredis for async I/O
- Minimal dependencies, prefer stdlib where possible
- Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query, Recharts for charts
- UUID fields from asyncpg must be converted to str via `_row_dict()` helpers
- asyncpg interval parameters must be Python `timedelta` objects, not SQL strings
- Recharts v3 callbacks: do NOT add explicit type annotations on `formatter`/`tickFormatter` — let TypeScript infer

## Common Pitfalls
- asyncpg expects `timedelta` for `$N::interval` params, not strings like `'7 days'`
- asyncpg expects UUID objects/strings for `$N::uuid` params — synthetic IDs like `pattern:AAPL:earnings:7d` will fail
- The `competitor_relationships` table uses UUID company IDs — queries must join through `companies` to match by ticker
- The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI
- Ingestion jobs require `source_id` from the `sources` table — don't just pass `ticker`

## Documentation
- Do NOT create large summary/success markdown files after each step
- Keep notes short, concise, and organized under `docs/notes/`
- If a note isn't useful for future reference, don't write it

## Documentation Maintenance on Feature Changes
When implementing a feature or fix that introduces an impactful change, update the relevant documentation as part of the same commit or task. "Impactful" means any change that affects how someone installs, deploys, configures, operates, or understands the system. Specifically:

- **New database migrations**: Update `docs/architecture-data-pipeline.md` or `docs/api-reference.md` if new tables, views, or endpoints are added. Update `project-context.md` steering file with the new migration number.
- **New API endpoints**: Update `docs/api-reference.md` with the endpoint path, method, parameters, and response shape.
- **New services or service changes**: Update `docs/architecture-docker-compose.md` and `docs/docker-deployment.md` if a new service is added or an existing service's configuration changes.
- **Helm chart changes**: Update `docs/helm-reference.md` if new values, services, or config options are added.
- **New environment variables or secrets**: Update `docs/LOCAL_DEV_SETUP.md` and the project-context steering file.
- **Install/deploy script changes**: Update `deploy-docker.sh`, `docs/docker-deployment.md`, or the relevant runme scripts if the deploy process changes.
- **Frontend route or page additions**: Update `docs/api-reference.md` (if it covers UI routes) and ensure the nav item is documented.
- **README.md**: Update the top-level `README.md` when a major new capability is added (new signal layer, new dashboard section, new trading feature).
- **Steering files**: Update `.kiro/steering/project-context.md` when migration numbers advance, new services are added, or key conventions change.

The goal is that someone reading the docs can always understand the current state of the system without reading the source code. When in doubt, update the doc.