phase 16: fix TS strict mode errors, node 24, update steering docs
This commit is contained in:
Celes Renata
@@ -1,28 +1,41 @@
|
||||
# Development Process — Test-Develop-Debug
|
||||
|
||||
## Local Environment
|
||||
- Python 3.12 via NixOS, virtualenv at `.venv/`
|
||||
- Always use `.venv/bin/python` or activate with `source .venv/bin/activate` before running Python commands
|
||||
- When running `pytest`, `ruff`, or any Python tool, use the `.venv` — e.g. `python -m pytest` (not bare `pytest` which may resolve to system Python)
|
||||
- Node.js 24 available for frontend work; `frontend/` has its own `node_modules/`
|
||||
|
||||
## Workflow
|
||||
1. Write or update tests for the target behavior
|
||||
2. Implement the minimal code to pass
|
||||
3. Debug failures, fix, re-run
|
||||
4. Commit and push after each phase completes
|
||||
5. GitHub Actions builds container images and pushes to GHCR
|
||||
6. Deploy to cluster via `kubectl apply`
|
||||
5. GitHub Actions CI automatically builds container images and pushes to GHCR
|
||||
6. Deploy to cluster via Helm or `kubectl apply`
|
||||
|
||||
## Testing
|
||||
- Use `pytest` with `pytest-asyncio` for async code
|
||||
- Tests live alongside service code or in a top-level `tests/` directory
|
||||
- Run tests with `pytest --tb=short -q` or `pytest -x` for fail-fast
|
||||
- Tests live in the top-level `tests/` directory
|
||||
- Run tests with `python -m pytest tests/ -x --tb=short -q`
|
||||
- Focus on core logic, not mocking infrastructure
|
||||
|
||||
## Build and Deploy
|
||||
- Always build and test Docker images locally before pushing to GitHub
|
||||
- Only push to GitHub after local build succeeds — don't waste CI credits on broken builds
|
||||
- Dockerfile at `docker/Dockerfile`
|
||||
- GitHub workflow at `.github/workflows/build.yml`
|
||||
- Images tagged as `ghcr.io/celesrenata/stonks-oracle/<service>:<sha>` and `:latest`
|
||||
- K8s manifests reference GHCR images
|
||||
- Deploy: `kubectl apply -f infra/k8s/`
|
||||
- Local build: `make build` → verify → `git push` → CI builds and pushes to GHCR
|
||||
## CI/CD — GitHub Actions
|
||||
- Workflow file: `.github/workflows/build.yml`
|
||||
- Triggers on push to `main` and PRs
|
||||
- Jobs:
|
||||
- `lint-and-test`: runs ruff lint + pytest on ubuntu with Python 3.12
|
||||
- `build-services`: matrix build of all Python services via `docker/Dockerfile`, pushes to GHCR with `:<sha>` and `:latest` tags
|
||||
- `build-dashboard`: builds `frontend/Dockerfile` separately, pushes `dashboard` image to GHCR
|
||||
- CI handles image building and pushing — do NOT manually `docker push` unless CI is broken or you need to bypass it
|
||||
- After pushing to `main`, wait for CI to complete before deploying (check GitHub Actions status)
|
||||
- If you need to build locally for testing: `make build` or `docker build` directly, but let CI do the GHCR push
|
||||
|
||||
## Deploy
|
||||
- Helm chart at `infra/helm/stonks-oracle/`
|
||||
- Deploy: `helm upgrade --install stonks-oracle infra/helm/stonks-oracle -n stonks-oracle`
|
||||
- Alternative raw manifests: `kubectl apply -f infra/k8s/`
|
||||
- To restart a deployment after CI pushes new images: `kubectl rollout restart deployment/<name> -n stonks-oracle`
|
||||
|
||||
## Git Conventions
|
||||
- Commit after each completed phase task
|
||||
@@ -35,10 +48,10 @@
|
||||
- FastAPI for HTTP services
|
||||
- asyncio + asyncpg/aioredis for async I/O
|
||||
- Minimal dependencies, prefer stdlib where possible
|
||||
- Frontend: React 19, TypeScript strict mode, Tailwind CSS, TanStack Router/Query
|
||||
|
||||
## Documentation
|
||||
- Do NOT create large summary/success markdown files after each step
|
||||
- Keep notes short, concise, and organized under `docs/notes/`
|
||||
- Name note files to match the task they relate to (e.g. `docs/notes/phase0-k8s-manifests.md`)
|
||||
- This makes them recallable by task without guessing
|
||||
- If a note isn't useful for future reference, don't write it
|
||||
|
||||
@@ -2,7 +2,13 @@
|
||||
|
||||
## Overview
|
||||
Stonks Oracle is a Kubernetes-native AI market intelligence and paper-trading platform.
|
||||
Python monorepo with services under `services/`, infrastructure under `infra/`, lakehouse schemas under `lakehouse/`, and dashboards under `dashboards/`.
|
||||
Python monorepo with services under `services/`, infrastructure under `infra/`, lakehouse schemas under `lakehouse/`, frontend React dashboard under `frontend/`, and dashboards under `dashboards/`.
|
||||
|
||||
## Local Dev Environment
|
||||
- NixOS dev environment, Python 3.12
|
||||
- Virtual environment at `.venv/` — always use it for Python commands
|
||||
- Node.js 24 for frontend (`frontend/` directory)
|
||||
- Docker available locally for image builds
|
||||
|
||||
## Infrastructure
|
||||
- Kubernetes cluster: 4x NixOS nodes (gremlin-1 through gremlin-4), reachable via `kubectl`, `virtctl`, `ssh root@gremlin-{1,2,3,4}`
|
||||
@@ -10,7 +16,14 @@ Python monorepo with services under `services/`, infrastructure under `infra/`,
|
||||
- Ingress: Traefik, domain `*.celestium.life`
|
||||
- Cert-Manager: `ca-issuer` (local CA) for internal services, `celestium-le-production` (Let's Encrypt) for public-facing
|
||||
- Container registry: `ghcr.io/celesrenata/stonks-oracle`
|
||||
- CI: GitHub Actions builds containers, cluster pulls from GHCR
|
||||
|
||||
## CI/CD
|
||||
- GitHub Actions workflow at `.github/workflows/build.yml`
|
||||
- Push to `main` triggers: lint → test → build all service images + dashboard image → push to GHCR
|
||||
- Images tagged as `ghcr.io/celesrenata/stonks-oracle/<service>:<sha>` and `:latest`
|
||||
- Dashboard image built from `frontend/Dockerfile` (multi-stage: node → nginx)
|
||||
- Python service images built from `docker/Dockerfile` with `SERVICE_CMD` build arg
|
||||
- Let CI handle image builds and pushes — only build locally for testing or when CI is unavailable
|
||||
|
||||
## Existing Cluster Services (do NOT redeploy these)
|
||||
- PostgreSQL: `postgresql-rw.postgresql-service.svc.cluster.local:5432`
|
||||
@@ -18,16 +31,10 @@ Python monorepo with services under `services/`, infrastructure under `infra/`,
|
||||
- MinIO: `minio.minio-service.svc.cluster.local:80` (API), console at `minio-crawler-console.minio-service.svc.cluster.local:9090`
|
||||
- Ollama: `ollama.ollama-service.svc.cluster.local:11434` (cluster-internal), also at `http://10.1.1.12:2701` (external), GPU: 4070 Ti Super 16GB
|
||||
|
||||
## Development Process
|
||||
- Test-Develop-Debug (TDD) cycle
|
||||
- After each phase: commit, push, build via GitHub Actions, deploy to cluster
|
||||
- Local builds for dev iteration, GitHub workflows for CI/CD
|
||||
- Python 3.12, NixOS dev environment
|
||||
|
||||
## Key Conventions
|
||||
- All services use `services/shared/config.py` for configuration via env vars
|
||||
- Redis queues defined in `services/shared/redis_keys.py`
|
||||
- Pydantic schemas in `services/shared/schemas.py`
|
||||
- K8s manifests in `infra/k8s/`, all in `stonks-oracle` namespace
|
||||
- K8s manifests in `infra/k8s/`, Helm chart in `infra/helm/stonks-oracle/`, all in `stonks-oracle` namespace
|
||||
- Lakehouse DDL in `lakehouse/schemas/`
|
||||
- Crawler patterns inspired by Noctipede (`~/sources/splinterstice/noctipede`): BeautifulSoup + requests with retry adapters, content hashing, boilerplate stripping, quality scoring
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
# Stage 1: Build
|
||||
FROM node:20-alpine AS build
|
||||
FROM node:24-alpine AS build
|
||||
WORKDIR /app
|
||||
COPY package.json package-lock.json ./
|
||||
RUN npm ci
|
||||
|
||||
@@ -35,8 +35,8 @@ export function OpsCoveragePage() {
|
||||
<tbody>
|
||||
{matrix.map((row, i) => (
|
||||
<tr key={i} className="border-b border-surface-700/50">
|
||||
<td className="px-3 py-2 font-mono font-semibold text-brand-300">{row.ticker as string}</td>
|
||||
<td className="px-3 py-2 text-gray-300">{row.legal_name as string}</td>
|
||||
<td className="px-3 py-2 font-mono font-semibold text-brand-300">{String(row.ticker)}</td>
|
||||
<td className="px-3 py-2 text-gray-300">{String(row.legal_name)}</td>
|
||||
<CoverageCell count={row.market_sources as number} />
|
||||
<CoverageCell count={row.news_sources as number} />
|
||||
<CoverageCell count={row.filings_sources as number} />
|
||||
@@ -61,7 +61,7 @@ export function OpsCoveragePage() {
|
||||
const missingTypes = expected.filter((t) => !activeTypes.includes(t));
|
||||
return (
|
||||
<div key={i} className="flex items-center gap-3 rounded border border-yellow-700/30 bg-yellow-900/10 p-2">
|
||||
<span className="font-mono font-semibold text-brand-300">{m.ticker as string}</span>
|
||||
<span className="font-mono font-semibold text-brand-300">{String(m.ticker)}</span>
|
||||
<span className="text-xs text-gray-500">missing:</span>
|
||||
{missingTypes.map((t) => (
|
||||
<StatusBadge key={t} status={t} />
|
||||
@@ -81,12 +81,12 @@ export function OpsCoveragePage() {
|
||||
{stale.map((s, i) => (
|
||||
<div key={i} className="flex items-center justify-between rounded border border-red-700/30 bg-red-900/10 p-2">
|
||||
<div className="flex items-center gap-3">
|
||||
<span className="font-mono font-semibold text-brand-300">{s.ticker as string}</span>
|
||||
<StatusBadge status={s.source_type as string} />
|
||||
<span className="text-xs text-gray-400">{s.source_name as string}</span>
|
||||
<span className="font-mono font-semibold text-brand-300">{String(s.ticker)}</span>
|
||||
<StatusBadge status={String(s.source_type)} />
|
||||
<span className="text-xs text-gray-400">{String(s.source_name)}</span>
|
||||
</div>
|
||||
<div className="text-xs text-gray-500">
|
||||
Last success: {s.last_success ? new Date(s.last_success as string).toLocaleString() : 'never'}
|
||||
Last success: {s.last_success ? new Date(String(s.last_success)).toLocaleString() : 'never'}
|
||||
{s.recent_failures ? ` | ${s.recent_failures} failures (24h)` : ''}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -45,11 +45,11 @@ export function OpsIngestionPage() {
|
||||
|
||||
{/* Summary stats */}
|
||||
<div className="grid grid-cols-2 gap-3 sm:grid-cols-5">
|
||||
<StatCard label="Total Runs" value={s.total_runs} />
|
||||
<StatCard label="Completed" value={s.completed} color="text-green-400" />
|
||||
<StatCard label="Failed" value={s.failed} color="text-red-400" />
|
||||
<StatCard label="Items Fetched" value={s.total_items_fetched} />
|
||||
<StatCard label="New Items" value={s.total_items_new} />
|
||||
<StatCard label="Total Runs" value={String(s.total_runs ?? '—')} />
|
||||
<StatCard label="Completed" value={String(s.completed ?? '—')} color="text-green-400" />
|
||||
<StatCard label="Failed" value={String(s.failed ?? '—')} color="text-red-400" />
|
||||
<StatCard label="Items Fetched" value={String(s.total_items_fetched ?? '—')} />
|
||||
<StatCard label="New Items" value={String(s.total_items_new ?? '—')} />
|
||||
</div>
|
||||
|
||||
{/* Throughput chart */}
|
||||
@@ -68,7 +68,7 @@ export function OpsIngestionPage() {
|
||||
</Card>
|
||||
|
||||
{/* By source type */}
|
||||
{s.by_source_type && (
|
||||
{s.by_source_type ? (
|
||||
<Card>
|
||||
<h2 className="mb-3 text-sm font-medium text-gray-400">By Source Type</h2>
|
||||
<div className="overflow-x-auto">
|
||||
@@ -85,7 +85,7 @@ export function OpsIngestionPage() {
|
||||
<tbody>
|
||||
{(s.by_source_type as Array<Record<string, unknown>>).map((row, i) => (
|
||||
<tr key={i} className="border-b border-surface-700/50">
|
||||
<td className="px-3 py-2 text-gray-300">{row.source_type as string}</td>
|
||||
<td className="px-3 py-2 text-gray-300">{String(row.source_type)}</td>
|
||||
<td className="px-3 py-2 text-gray-300">{String(row.runs)}</td>
|
||||
<td className="px-3 py-2 text-green-400">{String(row.completed)}</td>
|
||||
<td className="px-3 py-2 text-red-400">{String(row.failed)}</td>
|
||||
@@ -96,15 +96,15 @@ export function OpsIngestionPage() {
|
||||
</table>
|
||||
</div>
|
||||
</Card>
|
||||
)}
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function StatCard({ label, value, color = 'text-gray-100' }: { label: string; value: unknown; color?: string }) {
|
||||
function StatCard({ label, value, color = 'text-gray-100' }: { label: string; value: string; color?: string }) {
|
||||
return (
|
||||
<Card className="text-center">
|
||||
<div className={`text-xl font-bold ${color}`}>{value != null ? String(value) : '—'}</div>
|
||||
<div className={`text-xl font-bold ${color}`}>{value}</div>
|
||||
<div className="text-xs text-gray-500">{label}</div>
|
||||
</Card>
|
||||
);
|
||||
|
||||
@@ -20,7 +20,7 @@ export function OpsModelPage() {
|
||||
|
||||
{/* Key metrics */}
|
||||
<div className="grid grid-cols-2 gap-3 sm:grid-cols-5">
|
||||
<StatCard label="Total Extractions" value={p.total_extractions} />
|
||||
<StatCard label="Total Extractions" value={String(p.total_extractions ?? '—')} />
|
||||
<StatCard label="Success Rate" value={p.success_rate != null ? `${((p.success_rate as number) * 100).toFixed(1)}%` : '—'} color="text-green-400" />
|
||||
<StatCard label="Avg Latency" value={p.avg_duration_ms != null ? `${Math.round(p.avg_duration_ms as number)}ms` : '—'} />
|
||||
<StatCard label="Retry Rate" value={p.retry_rate != null ? `${((p.retry_rate as number) * 100).toFixed(1)}%` : '—'} color="text-yellow-400" />
|
||||
@@ -40,20 +40,20 @@ export function OpsModelPage() {
|
||||
<div key={i} className="rounded border border-surface-700 bg-surface-950 p-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="font-mono text-sm text-brand-300">{f.ticker as string}</span>
|
||||
<span className="font-mono text-sm text-brand-300">{String(f.ticker)}</span>
|
||||
<StatusBadge status="failed" />
|
||||
<span className="text-xs text-gray-500">{f.model_name as string}</span>
|
||||
<span className="text-xs text-gray-500">{String(f.model_name)}</span>
|
||||
</div>
|
||||
<span className="text-xs text-gray-500">{f.recorded_at ? new Date(f.recorded_at as string).toLocaleString() : ''}</span>
|
||||
<span className="text-xs text-gray-500">{f.recorded_at ? new Date(String(f.recorded_at)).toLocaleString() : ''}</span>
|
||||
</div>
|
||||
<div className="mt-1 text-xs text-gray-400">
|
||||
{f.document_title as string} ({f.document_type as string})
|
||||
{f.document_title ? String(f.document_title) : ''} ({String(f.document_type)})
|
||||
</div>
|
||||
{f.validation_errors && (
|
||||
{f.validation_errors ? (
|
||||
<div className="mt-1 text-xs text-red-400">
|
||||
{JSON.stringify(f.validation_errors)}
|
||||
{JSON.stringify(f.validation_errors) ?? ''}
|
||||
</div>
|
||||
)}
|
||||
) : null}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
@@ -63,10 +63,10 @@ export function OpsModelPage() {
|
||||
);
|
||||
}
|
||||
|
||||
function StatCard({ label, value, color = 'text-gray-100' }: { label: string; value: unknown; color?: string }) {
|
||||
function StatCard({ label, value, color = 'text-gray-100' }: { label: string; value: string; color?: string }) {
|
||||
return (
|
||||
<Card className="text-center">
|
||||
<div className={`text-xl font-bold ${color}`}>{value != null ? String(value) : '—'}</div>
|
||||
<div className={`text-xl font-bold ${color}`}>{value}</div>
|
||||
<div className="text-xs text-gray-500">{label}</div>
|
||||
</Card>
|
||||
);
|
||||
|
||||
@@ -51,9 +51,9 @@ export function OrderDetailPage() {
|
||||
<StatusBadge status={ev.event_type} />
|
||||
<div className="flex-1">
|
||||
<div className="text-xs text-gray-400">{new Date(ev.created_at).toLocaleString()}</div>
|
||||
{ev.data && (
|
||||
<pre className="mt-1 text-xs text-gray-500">{JSON.stringify(ev.data, null, 2)}</pre>
|
||||
)}
|
||||
{ev.data ? (
|
||||
<pre className="mt-1 text-xs text-gray-500">{JSON.stringify(ev.data, null, 2) ?? ''}</pre>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
@@ -68,9 +68,9 @@ export function OrderDetailPage() {
|
||||
<div className="space-y-1">
|
||||
{(order.audit_trail as Array<Record<string, unknown>>).map((entry, i) => (
|
||||
<div key={i} className="flex gap-3 text-xs">
|
||||
<span className="text-gray-500">{entry.created_at ? new Date(entry.created_at as string).toLocaleString() : ''}</span>
|
||||
<span className="text-gray-400">{entry.event_type as string}</span>
|
||||
<span className="text-gray-300">{entry.description as string}</span>
|
||||
<span className="text-gray-500">{entry.created_at ? new Date(String(entry.created_at)).toLocaleString() : ''}</span>
|
||||
<span className="text-gray-400">{String(entry.event_type)}</span>
|
||||
<span className="text-gray-300">{String(entry.description)}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
@@ -79,23 +79,23 @@ export function TrendDetailPage() {
|
||||
<div key={i} className="rounded-lg border border-surface-700 bg-surface-950 p-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<StatusBadge status={ev.evidence_type as string} />
|
||||
<span className="text-sm text-gray-200">{(ev.title as string) ?? 'Untitled'}</span>
|
||||
<StatusBadge status={String(ev.evidence_type)} />
|
||||
<span className="text-sm text-gray-200">{String(ev.title ?? 'Untitled')}</span>
|
||||
</div>
|
||||
<span className="font-mono text-xs text-gray-500">rank: {((ev.rank_score as number) ?? 0).toFixed(3)}</span>
|
||||
</div>
|
||||
<div className="mt-1 flex gap-4 text-xs text-gray-500">
|
||||
<span>{ev.document_type as string}</span>
|
||||
<span>{ev.source_type as string}</span>
|
||||
{ev.publisher && <span>{ev.publisher as string}</span>}
|
||||
{ev.published_at && <span>{new Date(ev.published_at as string).toLocaleDateString()}</span>}
|
||||
<span>{String(ev.document_type)}</span>
|
||||
<span>{String(ev.source_type)}</span>
|
||||
{ev.publisher ? <span>{String(ev.publisher)}</span> : null}
|
||||
{ev.published_at ? <span>{new Date(String(ev.published_at)).toLocaleDateString()}</span> : null}
|
||||
</div>
|
||||
{ev.intelligence && (
|
||||
{ev.intelligence ? (
|
||||
<div className="mt-2 text-xs text-gray-400">
|
||||
<span className="text-gray-500">Summary: </span>
|
||||
{((ev.intelligence as Record<string, unknown>).summary as string) ?? '—'}
|
||||
{String((ev.intelligence as Record<string, unknown>).summary ?? '—')}
|
||||
</div>
|
||||
)}
|
||||
) : null}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user