From f9284ce0a8f623c77d09e99d6668ff39bda7cc7b Mon Sep 17 00:00:00 2001 From: Celes Renata Date: Mon, 13 Jul 2026 09:12:50 +0000 Subject: [PATCH] docs: update steering files with CI DNS fix, specialist build, and new pitfalls --- .kiro/steering/development-process.md | 15 +++++++++++++++ .kiro/steering/project-context.md | 8 ++++++++ 2 files changed, 23 insertions(+) diff --git a/.kiro/steering/development-process.md b/.kiro/steering/development-process.md index 8b23987..c7721bd 100644 --- a/.kiro/steering/development-process.md +++ b/.kiro/steering/development-process.md @@ -35,12 +35,22 @@ - Push to Gitea: `git push gitea main` - Gitea remote: `http://admin:@10.1.1.12:30300/admin/stonks-oracle.git` - Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor +- Build pipelines split across `build-1.yml`, `build-2.yml`, `build-3.yml` for parallelism - ArgoCD watches Gitea `main` and auto-syncs beta/paper/live stages - **Do NOT push directly to GitHub** — GitHub is the promotion target after CI passes - Once Woodpecker builds and tests pass, code is promoted to GitHub (`git push origin main`) - CI handles all image builds and pushes — do NOT manually docker push - Check Woodpecker CI status from the Gitea web UI or Woodpecker dashboard +### Dashboard Build (npm ci in K8s) +- `build-3.yml` has a `npm-install-dashboard` step that runs `npm ci` in a `node:24-alpine` pod +- K8s CoreDNS causes `EAI_AGAIN` errors for Node.js under concurrent DNS lookups +- Fix: the step resolves `registry.npmjs.org` to IPv4 via Google DoH and pins it in `/etc/hosts` +- `NODE_OPTIONS=--dns-result-order=ipv4first` env var is set as additional safety +- `frontend/.dockerignore` must NOT exclude `node_modules` — the Dockerfile expects it pre-staged +- The subsequent `build-dashboard` step uses `frontend/` as Docker context (includes `node_modules`) +- If `npm ci` hangs: check `/etc/hosts` pinning worked, check `npm config set loglevel http` for which request is stuck + ## Deploy - Full deploy/redeploy: `bash ~/sources/kube/stonks-oracle/runmefirst.sh` (from gremlin-1) - Full teardown: `bash ~/sources/kube/stonks-oracle/runmelast.sh` (from gremlin-1) @@ -93,6 +103,11 @@ Ingestion jobs MUST include `source_id`, `source_type`, `ticker`, `company_id`, - The `competitor_relationships` table uses UUID company IDs — queries must join through `companies` to match by ticker - The dashboard Docker build uses TypeScript strict mode — unused imports that pass local diagnostics will fail in CI - Ingestion jobs require `source_id` from the `sources` table — don't just pass `ticker` +- `frontend/.dockerignore` must NOT contain `node_modules` — the CI pre-installs it and the Dockerfile relies on `COPY . .` including it +- `npm config set prefer-ip-address-family 4` does NOT exist in npm 10.x (Node 24) — don't use it +- Woodpecker `environment:` uses map syntax (`KEY: "value"`) not list syntax (`- KEY=value`) +- Node.js in Alpine K8s pods gets `EAI_AGAIN` from CoreDNS under load — pin hostnames in `/etc/hosts` for reliability +- Every Helm-deployed service MUST have a corresponding image build step in `.woodpecker/build-*.yml` - **Bash `!` in passwords/strings**: Bash interprets `!` inside double quotes as history expansion. NEVER use double quotes around strings containing `!`. Use single quotes instead: `'St0nks0racl3!'`. For kubectl exec with psql, use: `kubectl exec ... -- psql -U postgres -c "ALTER USER x WITH PASSWORD '"'"'password!'"'"';"` (single-quote escaping trick) ## No Premature Simplification diff --git a/.kiro/steering/project-context.md b/.kiro/steering/project-context.md index 12dbe48..b0049e8 100644 --- a/.kiro/steering/project-context.md +++ b/.kiro/steering/project-context.md @@ -46,12 +46,20 @@ Three-layer signal aggregation engine: - Pipeline stages: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor - Images tagged as `registry.celestium.life/stonks-oracle/:` and `:latest` - Dashboard image: `frontend/Dockerfile` (multi-stage: node:24 → nginx-unprivileged on port 8080) +- Dashboard build: `npm-install-dashboard` step in `build-3.yml` pre-installs `node_modules`, then `build-dashboard` runs the Docker build with `node_modules` in context - Superset image: `docker/Dockerfile.superset` (apache/superset + trino + psycopg2) - Python service images: `docker/Dockerfile` with `SERVICE_CMD` build arg +- Specialist image: built in `build-3.yml` like other Python services (`SERVICE_CMD=uvicorn services.specialist.app:app --host 0.0.0.0 --port 8000`) - Let CI handle image builds and pushes — do NOT manually `docker build && docker push` - **Do NOT push directly to GitHub** — GitHub (`origin`) is the promotion target after CI builds and tests pass - Promotion to GitHub: `git push origin main` (only after Woodpecker CI succeeds) +### CI DNS Workaround (npm) +- K8s CoreDNS causes `EAI_AGAIN` (temporary DNS failure) for Node.js/libuv under concurrent requests +- Fix: `npm-install-dashboard` step resolves `registry.npmjs.org` via Google DoH (`dns.google/resolve`) and pins the IPv4 address in `/etc/hosts` before running `npm ci` +- `NODE_OPTIONS=--dns-result-order=ipv4first` is set as env var (belt-and-suspenders) +- `frontend/.dockerignore` does NOT exclude `node_modules` (it must be in the Docker build context since the Dockerfile has no `RUN npm ci`) + ## Deployment Scripts - `~/sources/kube/stonks-oracle/runmefirst.sh` — full deploy: DB setup, migrations, Helm install, rolling restart (runs from gremlin-1 at 192.168.42.254 where secrets are available) - `~/sources/kube/stonks-oracle/runmelast.sh` — teardown: Helm uninstall, clean resources (preserves DB/MinIO/Redis)