Root cause: DNS returns IPv6 (AAAA) records for registry.npmjs.org
first. Alpine musl tries IPv6, which hangs because the cluster has
no IPv6 routing. NODE_OPTIONS=--dns-result-order=ipv4first forces
Node/npm to use IPv4 addresses.
BuildKit sandbox has no network access for DNS resolution. Instead of
fighting it, run npm ci in a normal Woodpecker step (which has full
network), then COPY the whole frontend dir (including node_modules)
into the Docker build context. The Dockerfile just runs npm run build.
Root cause confirmed: BuildKit sandbox creates isolated network namespace
that cannot resolve DNS even though the pod itself can. The buildx_flags
--driver-opt network=host only affects the buildkit daemon creation, not
the actual build steps. Adding network: host passes --network=host to
docker buildx build, making RUN steps use host networking.
Root cause: CoreDNS pods were failing to forward external DNS queries
(registry.npmjs.org.celestium.life -> no next plugin). Restarted
CoreDNS deployment. Reverted Dockerfile to original clean state.
The BuildKit sandbox cannot resolve registry.npmjs.org DNS. Setting
networkMode=host in the worker config forces all RUN steps to use
the host network stack, which has confirmed internet access.
BuildKit sandbox DNS cannot resolve registry.npmjs.org. Using
--network=host on the RUN step lets npm use the host network stack
directly, bypassing the broken sandbox DNS.
BuildKit sandbox does not inherit cluster DNS, causing npm ci to fail
with EAI_AGAIN. Inject Google/Cloudflare DNS into resolv.conf before
running npm ci.
All pipeline files now specify clone.git.settings.remote with the
Gitea service ClusterIP (10.43.73.77:3000) instead of relying on
DNS resolution of gitea-service.git-server.svc.cluster.local.
This permanently fixes the intermittent DNS resolution failures
that occur when multiple build pipelines clone simultaneously.
Document Stages now shows time-filtered counts from /health endpoint
(consistent with Parsing/Extraction/Trends sections) with all-time
totals shown as a subtle subtitle from the SSE stream. Added "(last Xh)"
labels to all sections so the time window context is always visible.
_process_macro_classification never updated document status, leaving
macro_event docs stuck in parsed forever. Now marks them extracted on
success or extraction_failed on error.
- Replace setdefault() with explicit None checks in _normalize_extraction_data()
- Coerce null top-level fields (summary, novelty_score, confidence) to defaults
- Coerce null company fields (ticker, impact_score, impact_horizon, etc.) to defaults
- Filter out company entries with empty ticker after normalization
- Prevents schema validation failures when model returns null for required fields
- Strategy: Recreate (GPU can only be used by one pod at a time)
- nuextract-external targetPort: 8080 (nginx proxy port)
- served-model-name: numind/NuExtract3
- DB ai_agents updated to model=numind/NuExtract3 provider=vllm
- Ollama scaled to 0 — vLLM gets entire 16GB GPU
- NuExtract3 (9B) running at 80% GPU utilization
- Both OLLAMA_BASE_URL and VLLM_BASE_URL point to vLLM service
- Fixed gremlin-1 missing default route (no internet access)
- Removed GPU resource limit (time-slicing handles allocation)
- Removed init container (direct download works with internet fix)
- vLLM image mirrored to registry.celestium.life/stonks-oracle/vllm-openai
- Deployment uses Harbor image (Docker Hub IPv6 unreachable from cluster)
- All 3 pipelines use vllm-external.vllm-service.svc.cluster.local:2701
- K8s manifests at infra/kube-vllm/ and synced to ~/sources/kube/vllm
K8s vLLM deployment blocked by Docker Hub IPv6 connectivity issue.
Falling back to NixOS systemd vLLM service on esnixi.
K8s manifests remain in infra/kube-vllm/ for future use.
Sentiment Heatmap and Prediction Accuracy passed limit=500 but
the trends and recommendations endpoints enforce limit<=200,
causing a 422 validation error that made both panels show blank.
Celes Renata
Celes Hillyerd