stonks-oracle/.kiro/steering/development-process.md

Development Process — Test-Develop-Debug

Local Environment

• Ubuntu dev machine, Python 3.12, virtualenv at .venv/
• Always use .venv/bin/python or activate with source .venv/bin/activate before running Python commands
• Node.js 24 via nvm — always load nvm before running Node/npm/npx commands: export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" && nvm use 24
• For tools not available in .venv/ (ruff, gh, etc.), install via pip or apt as needed
• Frontend tests: load nvm first, then cd frontend && npx vitest --run
• Python tests: .venv/bin/ruff check services/ then .venv/bin/python -m pytest tests/ -x --tb=short -q

Workflow

1. Write or update tests for the target behavior
2. Implement the minimal code to pass
3. Debug failures, fix, re-run
4. Commit and push — CI builds images automatically
5. Deploy: helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle
6. Restart changed services: kubectl rollout restart deployment/<name> -n stonks-oracle

Testing

• Python: pytest with pytest-asyncio for async code, tests in tests/
• Property-based tests: Hypothesis with @settings(max_examples=100), files prefixed test_pbt_*
• Frontend: Vitest + MSW (Mock Service Worker) for deterministic API mocking, tests in frontend/src/test/
• Run Python tests: python -m pytest tests/ -x --tb=short -q
• Run frontend tests: cd frontend && npx vitest --run
• Lint Python: .venv/bin/ruff check services/
• Always run .venv/bin/ruff check services/ before committing Python changes — CI will reject the push if ruff fails
• Ruff auto-fix: .venv/bin/ruff check --fix services/ (fixes import sorting and other auto-fixable issues)
• Ruff is pinned to ruff==0.15.10 in requirements.txt — CI uses the same version
• Ruff config: ruff.toml with known-first-party = ["services"] for consistent import sorting
• Pre-existing test failures (not regressions): test_extractor_prompts.py, test_extractor_schemas.py, test_filings_adapter.py, test_ollama_client.py

CI/CD — Woodpecker CI (Gitea) → GitHub promotion

• Woodpecker pipelines in .woodpecker/ — triggered by push to main on Gitea
• Push to Gitea: git push gitea main
• Gitea remote: http://admin:<password>@10.1.1.12:30300/admin/stonks-oracle.git
• Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
• ArgoCD watches Gitea main and auto-syncs beta/paper/live stages
• Do NOT push directly to GitHub — GitHub is the promotion target after CI passes
• Once Woodpecker builds and tests pass, code is promoted to GitHub (git push origin main)
• CI handles all image builds and pushes — do NOT manually docker push
• Check Woodpecker CI status from the Gitea web UI or Woodpecker dashboard

Deploy

• Full deploy/redeploy: bash ~/sources/kube/stonks-oracle/runmefirst.sh (from gremlin-1)
• Full teardown: bash ~/sources/kube/stonks-oracle/runmelast.sh (from gremlin-1)
• Quick Helm upgrade: helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle
• Restart single service: kubectl rollout restart deployment/<name> -n stonks-oracle
• Restart multiple: kubectl rollout restart deployment/aggregation deployment/query-api deployment/dashboard -n stonks-oracle
• Check pods: kubectl get pods -n stonks-oracle
• Check logs: kubectl logs deployment/<name> -n stonks-oracle --tail=30

Manually Triggering Ingestion

The scheduler runs on a cadence, but to trigger immediately:

# From a scheduler pod:
kubectl exec -n stonks-oracle <scheduler-pod> -- python -c "
import redis, json, asyncio, asyncpg
async def enqueue():
    pool = await asyncpg.create_pool(dsn='postgresql://stonks:<password>@postgresql-rw.postgresql-service.svc.cluster.local:5432/stonks')
    r = redis.from_url('redis://:<password>@redis-master.redis-service.svc.cluster.local:6379/0')
    rows = await pool.fetch('SELECT s.id AS source_id, s.source_type, s.config, c.id AS company_id, c.ticker FROM sources s JOIN companies c ON c.id = s.company_id WHERE s.active = TRUE AND c.active = TRUE')
    for row in rows:
        cfg = row['config'] if isinstance(row['config'], dict) else {}
        r.rpush('stonks:queue:ingestion', json.dumps({'source_id': str(row['source_id']), 'source_type': row['source_type'], 'ticker': row['ticker'], 'company_id': str(row['company_id']), 'config': cfg}))
    await pool.close()
asyncio.run(enqueue())
"

Ingestion jobs MUST include source_id, source_type, ticker, company_id, and config.

Git Conventions

• Commit after each completed phase task
• Commit message format: feat:, fix:, phase N: prefix
• Always push to Gitea: git push gitea main
• Do NOT push to GitHub (origin) directly — GitHub is the promotion target after CI passes
• ArgoCD syncs from Gitea automatically

Code Style

• Python 3.12, type hints everywhere
• Pydantic for data validation
• FastAPI for HTTP services
• asyncio + asyncpg/aioredis for async I/O
• Minimal dependencies, prefer stdlib where possible
• Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query, Recharts for charts
• UUID fields from asyncpg must be converted to str via _row_dict() helpers
• asyncpg interval parameters must be Python timedelta objects, not SQL strings
• Recharts v3 callbacks: do NOT add explicit type annotations on formatter/tickFormatter — let TypeScript infer

Common Pitfalls

• asyncpg expects timedelta for $N::interval params, not strings like '7 days'
• asyncpg expects UUID objects/strings for $N::uuid params — synthetic IDs like pattern:AAPL:earnings:7d will fail
• The competitor_relationships table uses UUID company IDs — queries must join through companies to match by ticker
• The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI
• Ingestion jobs require source_id from the sources table — don't just pass ticker

No Premature Simplification

Do NOT "simplify" code on impulse. When the urge arises to simplify a section, STOP and do this instead:

1. Evaluate the section: Read the full function/module, not just the part that looks complex.
2. Map the dependencies: Identify every caller, every consumer, every downstream component that depends on this code's behavior, return shape, or side effects.
3. Assess blast radius: Would changing this function break other implementations? Check imports, tests, API contracts, database queries, and frontend expectations.
4. Respect intentional complexity: If the code is complex because the domain is complex (financial math, multi-layer signal aggregation, Bayesian shrinkage), the complexity is load-bearing. Simplifying it will introduce bugs.
5. Only simplify when: The complexity is accidental (dead code, redundant branches, copy-paste artifacts) AND you have confirmed no downstream dependencies break.

This codebase has interconnected layers (ingestion → extraction → aggregation → recommendation → trading → validation). A "simple" change to a scoring function can cascade through trend summaries, recommendations, snapshot capture, and outcome evaluation. Always trace the full path before refactoring.

Documentation

• Do NOT create large summary/success markdown files after each step
• Keep notes short, concise, and organized under docs/notes/
• If a note isn't useful for future reference, don't write it

Documentation Maintenance on Feature Changes

When implementing a feature or fix that introduces an impactful change, update the relevant documentation as part of the same commit or task. "Impactful" means any change that affects how someone installs, deploys, configures, operates, or understands the system. Specifically:

• New database migrations: Update docs/architecture-data-pipeline.md or docs/api-reference.md if new tables, views, or endpoints are added. Update project-context.md steering file with the new migration number.
• New API endpoints: Update docs/api-reference.md with the endpoint path, method, parameters, and response shape.
• New services or service changes: Update docs/architecture-docker-compose.md and docs/docker-deployment.md if a new service is added or an existing service's configuration changes.
• Helm chart changes: Update docs/helm-reference.md if new values, services, or config options are added.
• New environment variables or secrets: Update docs/LOCAL_DEV_SETUP.md and the project-context steering file.
• Install/deploy script changes: Update deploy-docker.sh, docs/docker-deployment.md, or the relevant runme scripts if the deploy process changes.
• Frontend route or page additions: Update docs/api-reference.md (if it covers UI routes) and ensure the nav item is documented.
• README.md: Update the top-level README.md when a major new capability is added (new signal layer, new dashboard section, new trading feature).
• Steering files: Update .kiro/steering/project-context.md when migration numbers advance, new services are added, or key conventions change.

The goal is that someone reading the docs can always understand the current state of the system without reading the source code. When in doubt, update the doc.