admin/stonks-oracle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update steering files with CI DNS fix, specialist build, and new pitfalls

Browse Source
This commit is contained in:
Celes Renata
2026-07-13 09:12:50 +00:00
parent d4f1b529a5
commit f9284ce0a8
2 changed files with 23 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/steering/development-process.md
+15
View File
@@ -35,12 +35,22 @@
- Push to Gitea: `git push gitea main` - Push to Gitea: `git push gitea main`
- Gitea remote: `http://admin:<password>@10.1.1.12:30300/admin/stonks-oracle.git` - Gitea remote: `http://admin:<password>@10.1.1.12:30300/admin/stonks-oracle.git`
- Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor - Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
- Build pipelines split across `build-1.yml`, `build-2.yml`, `build-3.yml` for parallelism
- ArgoCD watches Gitea `main` and auto-syncs beta/paper/live stages - ArgoCD watches Gitea `main` and auto-syncs beta/paper/live stages
- **Do NOT push directly to GitHub** — GitHub is the promotion target after CI passes - **Do NOT push directly to GitHub** — GitHub is the promotion target after CI passes
- Once Woodpecker builds and tests pass, code is promoted to GitHub (`git push origin main`) - Once Woodpecker builds and tests pass, code is promoted to GitHub (`git push origin main`)
- CI handles all image builds and pushes — do NOT manually docker push - CI handles all image builds and pushes — do NOT manually docker push
- Check Woodpecker CI status from the Gitea web UI or Woodpecker dashboard - Check Woodpecker CI status from the Gitea web UI or Woodpecker dashboard
### Dashboard Build (npm ci in K8s)
- `build-3.yml` has a `npm-install-dashboard` step that runs `npm ci` in a `node:24-alpine` pod
- K8s CoreDNS causes `EAI_AGAIN` errors for Node.js under concurrent DNS lookups
- Fix: the step resolves `registry.npmjs.org` to IPv4 via Google DoH and pins it in `/etc/hosts`
- `NODE_OPTIONS=--dns-result-order=ipv4first` env var is set as additional safety
- `frontend/.dockerignore` must NOT exclude `node_modules` — the Dockerfile expects it pre-staged
- The subsequent `build-dashboard` step uses `frontend/` as Docker context (includes `node_modules`)
- If `npm ci` hangs: check `/etc/hosts` pinning worked, check `npm config set loglevel http` for which request is stuck
## Deploy ## Deploy
- Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1) - Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1)
- Full teardown: `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1) - Full teardown: `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1)
@@ -93,6 +103,11 @@ Ingestion jobs MUST include `source_id`, `source_type`, `ticker`, `company_id`,
- The `competitor_relationships` table uses UUID company IDs — queries must join through `companies` to match by ticker - The `competitor_relationships` table uses UUID company IDs — queries must join through `companies` to match by ticker
- The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI - The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI
- Ingestion jobs require `source_id` from the `sources` table — don't just pass `ticker` - Ingestion jobs require `source_id` from the `sources` table — don't just pass `ticker`
- `frontend/.dockerignore` must NOT contain `node_modules` — the CI pre-installs it and the Dockerfile relies on `COPY . .` including it
- `npm config set prefer-ip-address-family 4` does NOT exist in npm 10.x (Node 24) — don't use it
- Woodpecker `environment:` uses map syntax (`KEY: "value"`) not list syntax (`- KEY=value`)
- Node.js in Alpine K8s pods gets `EAI_AGAIN` from CoreDNS under load — pin hostnames in `/etc/hosts` for reliability
- Every Helm-deployed service MUST have a corresponding image build step in `.woodpecker/build-*.yml`
- **Bash `!` in passwords/strings**: Bash interprets `!` inside double quotes as history expansion. NEVER use double quotes around strings containing `!`. Use single quotes instead: `'St0nks0racl3!'`. For kubectl exec with psql, use: `kubectl exec ... -- psql -U postgres -c "ALTER USER x WITH PASSWORD '"'"'password!'"'"';"` (single-quote escaping trick) - **Bash `!` in passwords/strings**: Bash interprets `!` inside double quotes as history expansion. NEVER use double quotes around strings containing `!`. Use single quotes instead: `'St0nks0racl3!'`. For kubectl exec with psql, use: `kubectl exec ... -- psql -U postgres -c "ALTER USER x WITH PASSWORD '"'"'password!'"'"';"` (single-quote escaping trick)
## No Premature Simplification ## No Premature Simplification
.kiro/steering/project-context.md
+8
View File
@@ -46,12 +46,20 @@ Three-layer signal aggregation engine:
- Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor - Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
- Images tagged as `registry.celestium.life/stonks-oracle/<service>:<sha>` and `:latest` - Images tagged as `registry.celestium.life/stonks-oracle/<service>:<sha>` and `:latest`
- Dashboard image: `frontend/Dockerfile` (multi-stage: node:24 → nginx-unprivileged on port 8080) - Dashboard image: `frontend/Dockerfile` (multi-stage: node:24 → nginx-unprivileged on port 8080)
- Dashboard build: `npm-install-dashboard` step in `build-3.yml` pre-installs `node_modules`, then `build-dashboard` runs the Docker build with `node_modules` in context
- Superset image: `docker/Dockerfile.superset` (apache/superset + trino + psycopg2) - Superset image: `docker/Dockerfile.superset` (apache/superset + trino + psycopg2)
- Python service images: `docker/Dockerfile` with `SERVICE_CMD` build arg - Python service images: `docker/Dockerfile` with `SERVICE_CMD` build arg
- Specialist image: built in `build-3.yml` like other Python services (`SERVICE_CMD=uvicorn services.specialist.app:app --host 0.0.0.0 --port 8000`)
- Let CI handle image builds and pushes — do NOT manually `docker build && docker push` - Let CI handle image builds and pushes — do NOT manually `docker build && docker push`
- **Do NOT push directly to GitHub** — GitHub (`origin`) is the promotion target after CI builds and tests pass - **Do NOT push directly to GitHub** — GitHub (`origin`) is the promotion target after CI builds and tests pass
- Promotion to GitHub: `git push origin main` (only after Woodpecker CI succeeds) - Promotion to GitHub: `git push origin main` (only after Woodpecker CI succeeds)
### CI DNS Workaround (npm)
- K8s CoreDNS causes `EAI_AGAIN` (temporary DNS failure) for Node.js/libuv under concurrent requests
- Fix: `npm-install-dashboard` step resolves `registry.npmjs.org` via Google DoH (`dns.google/resolve`) and pins the IPv4 address in `/etc/hosts` before running `npm ci`
- `NODE_OPTIONS=--dns-result-order=ipv4first` is set as env var (belt-and-suspenders)
- `frontend/.dockerignore` does NOT exclude `node_modules` (it must be in the Docker build context since the Dockerfile has no `RUN npm ci`)
## Deployment Scripts ## Deployment Scripts
- `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available) - `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available)
- `~/sources/kube/stonks-oracle/runmelast.sh` — teardown: Helm uninstall, clean resources (preserves DB/MinIO/Redis) - `~/sources/kube/stonks-oracle/runmelast.sh` — teardown: Helm uninstall, clean resources (preserves DB/MinIO/Redis)