phase 0+1: project scaffold, k8s manifests, CI pipeline, steering, hooks, tests

- Repository structure for all services, infra, lakehouse, dashboards - K8s manifests targeting stonks-oracle namespace with GHCR images - Ingress via Traefik with ca-issuer TLS for internal services - ConfigMap wired to existing cluster services (pg, redis, minio, ollama) - GitHub Actions workflow for lint, test, multi-service container builds - Dockerfile with build-arg CMD per service - Makefile for local build/push/deploy - Steering rules for TDD workflow, K8s conventions, project context - Agent hooks for lint-on-save, test-on-save, k8s-validate, phase-commit - Ruff linter config, all lint issues fixed - 14 passing tests for schemas, config, redis keys - PostgreSQL migrations, Trino catalogs, Superset config, MinIO lifecycle
2026-04-11 03:25:08 -07:00
parent 8cfc4f423b
commit ebea70573b
90 changed files with 3590 additions and 19 deletions
@@ -0,0 +1,44 @@
+# Development Process — Test-Develop-Debug
+
+## Workflow
+1. Write or update tests for the target behavior
+2. Implement the minimal code to pass
+3. Debug failures, fix, re-run
+4. Commit and push after each phase completes
+5. GitHub Actions builds container images and pushes to GHCR
+6. Deploy to cluster via `kubectl apply`
+
+## Testing
+- Use `pytest` with `pytest-asyncio` for async code
+- Tests live alongside service code or in a top-level `tests/` directory
+- Run tests with `pytest --tb=short -q` or `pytest -x` for fail-fast
+- Focus on core logic, not mocking infrastructure
+
+## Build and Deploy
+- Always build and test Docker images locally before pushing to GitHub
+- Only push to GitHub after local build succeeds — don't waste CI credits on broken builds
+- Dockerfile at `docker/Dockerfile`
+- GitHub workflow at `.github/workflows/build.yml`
+- Images tagged as `ghcr.io/celesrenata/stonks-oracle/<service>:<sha>` and `:latest`
+- K8s manifests reference GHCR images
+- Deploy: `kubectl apply -f infra/k8s/`
+- Local build: `make build` → verify → `git push` → CI builds and pushes to GHCR
+
+## Git Conventions
+- Commit after each completed phase task
+- Commit message format: `phase N: short description`
+- Push to `main` branch triggers CI
+
+## Code Style
+- Python 3.12, type hints everywhere
+- Pydantic for data validation
+- FastAPI for HTTP services
+- asyncio + asyncpg/aioredis for async I/O
+- Minimal dependencies, prefer stdlib where possible
+
+## Documentation
+- Do NOT create large summary/success markdown files after each step
+- Keep notes short, concise, and organized under `docs/notes/`
+- Name note files to match the task they relate to (e.g. `docs/notes/phase0-k8s-manifests.md`)
+- This makes them recallable by task without guessing
+- If a note isn't useful for future reference, don't write it
@@ -0,0 +1,33 @@
+---
+inclusion: fileMatch
+fileMatchPattern: "infra/k8s/**"
+---
+# Kubernetes Conventions
+
+## Namespace
+All Stonks Oracle workloads deploy to `stonks-oracle` namespace.
+
+## TLS
+- Internal services: use `ca-issuer` ClusterIssuer (local CA)
+- Public-facing services (Superset, Query API): use `celestium-le-production` ClusterIssuer (Let's Encrypt)
+- Annotate ingress with `cert-manager.io/cluster-issuer`
+
+## Ingress
+- Traefik ingress controller
+- Domain pattern: `<service>.celestium.life`
+- Always create both HTTP and HTTPS ingress rules
+
+## Service References
+- PostgreSQL: `postgresql-rw.postgresql-service.svc.cluster.local:5432`
+- Redis: `redis-master.redis-service.svc.cluster.local:6379`
+- MinIO API: `minio.minio-service.svc.cluster.local:80`
+- Ollama: `ollama.ollama-service.svc.cluster.local:11434`
+
+## Images
+- All images from `ghcr.io/celesrenata/stonks-oracle/<service>:latest`
+- Use `imagePullPolicy: Always` in production
+- Use `imagePullSecrets` referencing `ghcr-secret` if repo is private
+
+## Labels
+- `app.kubernetes.io/part-of: stonks-oracle`
+- `app: <service-name>`
@@ -0,0 +1,33 @@
+# Stonks Oracle — Project Context
+
+## Overview
+Stonks Oracle is a Kubernetes-native AI market intelligence and paper-trading platform.
+Python monorepo with services under `services/`, infrastructure under `infra/`, lakehouse schemas under `lakehouse/`, and dashboards under `dashboards/`.
+
+## Infrastructure
+- Kubernetes cluster: 4x NixOS nodes (gremlin-1 through gremlin-4), reachable via `kubectl`, `virtctl`, `ssh root@gremlin-{1,2,3,4}`
+- NixOS configs stored at `/etc/nixos` on gremlin-1, git-pushed to other hosts
+- Ingress: Traefik, domain `*.celestium.life`
+- Cert-Manager: `ca-issuer` (local CA) for internal services, `celestium-le-production` (Let's Encrypt) for public-facing
+- Container registry: `ghcr.io/celesrenata/stonks-oracle`
+- CI: GitHub Actions builds containers, cluster pulls from GHCR
+
+## Existing Cluster Services (do NOT redeploy these)
+- PostgreSQL: `postgresql-rw.postgresql-service.svc.cluster.local:5432`
+- Redis: `redis-master.redis-service.svc.cluster.local:6379`
+- MinIO: `minio.minio-service.svc.cluster.local:80` (API), console at `minio-crawler-console.minio-service.svc.cluster.local:9090`
+- Ollama: `ollama.ollama-service.svc.cluster.local:11434` (cluster-internal), also at `http://10.1.1.12:2701` (external), GPU: 4070 Ti Super 16GB
+
+## Development Process
+- Test-Develop-Debug (TDD) cycle
+- After each phase: commit, push, build via GitHub Actions, deploy to cluster
+- Local builds for dev iteration, GitHub workflows for CI/CD
+- Python 3.12, NixOS dev environment
+
+## Key Conventions
+- All services use `services/shared/config.py` for configuration via env vars
+- Redis queues defined in `services/shared/redis_keys.py`
+- Pydantic schemas in `services/shared/schemas.py`
+- K8s manifests in `infra/k8s/`, all in `stonks-oracle` namespace
+- Lakehouse DDL in `lakehouse/schemas/`
+- Crawler patterns inspired by Noctipede (`~/sources/splinterstice/noctipede`): BeautifulSoup + requests with retry adapters, content hashing, boilerplate stripping, quality scoring