583 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Celes Renata	c43865cc9c	ci: verify Woodpecker OAuth reconnection	2026-07-21 21:57:45 +00:00
Celes Renata	987b32d2ff	fix: reduce correlation window 30d→7d to prevent OOM (132k→~30k rows)	2026-07-21 21:46:54 +00:00
Celes Renata	e1d1c70d59	ci: trigger pipeline after Woodpecker OAuth fix	2026-07-21 15:36:17 +00:00
Celes Renata	4f021511fa	fix: move correlation matrix to background task — prevents startup timeout/OOM	2026-07-21 15:23:22 +00:00
Celes Renata	6bde9d2726	fix: increase trading-engine memory limit 512Mi→1024Mi (OOMKilled by correlation matrix)	2026-07-21 15:20:38 +00:00
Celes Renata	620a8f0d91	fix: add specialist health check, update default namespace to stonks-oracle	2026-07-13 09:17:23 +00:00
Celes Renata	f9284ce0a8	docs: update steering files with CI DNS fix, specialist build, and new pitfalls	2026-07-13 09:12:50 +00:00
Celes Renata	d4f1b529a5	fix: add missing specialist image build step to CI pipeline	2026-07-13 08:08:23 +00:00
Celes Renata	9617b088bd	fix: allow node_modules in Docker context (pre-staged by CI step)	2026-07-13 07:40:19 +00:00
Celes Renata	c0d7f2f438	fix: pin registry.npmjs.org IPv4 in /etc/hosts to bypass broken DNS	2026-07-13 07:31:38 +00:00
Celes Renata	7229273461	fix: reduce npm maxsockets to 3, add retries for EAI_AGAIN DNS failures	2026-07-13 07:27:51 +00:00
Celes Renata	d18a63aab5	fix: remove invalid npm config, keep http logging for diagnostics	2026-07-13 07:23:40 +00:00
Celes Renata	7df45f7e27	fix: use map format for Woodpecker environment vars	2026-07-13 07:19:47 +00:00
Celes Renata	0d4a812e3d	fix: add DNS diagnostics to npm-install-dashboard CI step	2026-07-13 06:58:43 +00:00
Celes Renata	2fff553518	fix: force IPv4 for npm — cluster has no IPv6 egress ... Root cause: DNS returns IPv6 (AAAA) records for registry.npmjs.org first. Alpine musl tries IPv6, which hangs because the cluster has no IPv6 routing. NODE_OPTIONS=--dns-result-order=ipv4first forces Node/npm to use IPv4 addresses.	2026-07-13 06:41:46 +00:00
Celes Renata	2d516a725d	fix: pre-stage npm ci outside BuildKit sandbox ... BuildKit sandbox has no network access for DNS resolution. Instead of fighting it, run npm ci in a normal Woodpecker step (which has full network), then COPY the whole frontend dir (including node_modules) into the Docker build context. The Dockerfile just runs npm run build.	2026-07-13 05:55:41 +00:00
Celes Renata	cdad58ae28	fix: add network=host to dashboard buildx build step ... Root cause confirmed: BuildKit sandbox creates isolated network namespace that cannot resolve DNS even though the pod itself can. The buildx_flags --driver-opt network=host only affects the buildkit daemon creation, not the actual build steps. Adding network: host passes --network=host to docker buildx build, making RUN steps use host networking.	2026-07-13 05:52:00 +00:00
Celes Renata	9645d9c261	fix: revert Dockerfile hacks, restart CoreDNS to fix DNS resolution ... Root cause: CoreDNS pods were failing to forward external DNS queries (registry.npmjs.org.celestium.life -> no next plugin). Restarted CoreDNS deployment. Reverted Dockerfile to original clean state.	2026-07-13 05:20:48 +00:00
Celes Renata	05e6f21a16	fix: set BuildKit worker networkMode=host for dashboard build ... The BuildKit sandbox cannot resolve registry.npmjs.org DNS. Setting networkMode=host in the worker config forces all RUN steps to use the host network stack, which has confirmed internet access.	2026-07-13 05:02:35 +00:00
Celes Renata	d762a2b76f	fix: use RUN --network=host for npm ci in BuildKit ... BuildKit sandbox DNS cannot resolve registry.npmjs.org. Using --network=host on the RUN step lets npm use the host network stack directly, bypassing the broken sandbox DNS.	2026-07-13 04:47:37 +00:00
Celes Renata	46d7699ed0	fix: add explicit DNS resolvers to frontend Dockerfile for BuildKit ... BuildKit sandbox does not inherit cluster DNS, causing npm ci to fail with EAI_AGAIN. Inject Google/Cloudflare DNS into resolv.conf before running npm ci.	2026-07-13 04:38:59 +00:00
Celes Renata	13729bda98	ci: retry (DNS resolution failure in BuildKit)	2026-07-13 04:19:28 +00:00
Celes Renata	b5deeaa0d5	ci: retry build (npm ci hung on previous attempt)	2026-07-13 03:07:01 +00:00
Celes Renata	0aa72cbce1	fix: add numpy and scikit-learn to requirements for v3 calibrators	2026-07-13 02:19:44 +00:00
Celes Renata	a72f336ad1	feat: Intelligence Pipeline v3 — full implementation ... Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.	2026-07-13 02:14:59 +00:00
Celes Renata	84634a365e	feat: deep pipeline health check script ... Checks 12 aspects of the running deployment: - Pod health (replicas, restarts, OOMKills) - PostgreSQL connectivity & stats - Redis connectivity & queue backlog - MinIO connectivity - vLLM model availability - Alpaca broker account status - Pipeline flow (ingestion→extraction→aggregation→recommendation) - Model quality gate freshness - Paper trading portfolio status - API endpoint health - Resource usage - End-to-end latency (doc processing, trend/rec freshness) Usage: ./scripts/pipeline_health_check.sh [namespace]	2026-07-11 22:35:04 +00:00
Celes Renata	2f966b1caf	fix: scheduler memory 128Mi→256Mi, fix backfill script SQL param bug	2026-07-11 17:23:52 +00:00
Celes Renata	81ed2f0803	fix(ci): bypass DNS for clone — use Gitea ClusterIP directly ... All pipeline files now specify clone.git.settings.remote with the Gitea service ClusterIP (10.43.73.77:3000) instead of relying on DNS resolution of gitea-service.git-server.svc.cluster.local. This permanently fixes the intermittent DNS resolution failures that occur when multiple build pipelines clone simultaneously.	2026-07-11 05:36:41 +00:00
Celes Renata	5bd5c521dc	ci: dns debug	2026-07-11 05:28:31 +00:00
Celes Renata	9cbb559ac7	ci: retry - coredns permanent fix (dnsPolicy:None + forward 192.168.42.1)	2026-07-11 04:39:28 +00:00
Celes Renata	b13232a21f	ci: retry after reverting coredns and woodpecker dns to original config	2026-07-11 04:36:20 +00:00
Celes Renata	0cd052a0a7	ci: retry - restored search domains, ndots:2, timeout:10, attempts:5	2026-07-11 04:33:41 +00:00
Celes Renata	a2f4398f8d	ci: retry - dns config with ndots:3, timeout:5, attempts:5	2026-07-11 04:32:09 +00:00
Celes Renata	76e06d5951	ci: retry after scaling coredns to 4 replicas across all nodes	2026-07-10 23:25:16 +00:00
Celes Renata	96eabf595b	ci: retry after adding 192.168.42.1 to woodpecker DNS config	2026-07-10 23:21:18 +00:00
Celes Renata	430554fe0c	ci: retry after coredns fix + woodpecker agent restart	2026-07-10 23:18:59 +00:00
Celes Renata	853b85a2af	ci: retry after coredns forward fix	2026-07-10 23:14:18 +00:00
Celes Renata	1d104187f4	ci: retry after transient DNS failure	2026-07-10 23:11:32 +00:00
Celes Renata	d018c75182	fix: OpsPipeline dashboard — use time-filtered doc stages, add time labels ... Document Stages now shows time-filtered counts from /health endpoint (consistent with Parsing/Extraction/Trends sections) with all-time totals shown as a subtle subtitle from the SSE stream. Added "(last Xh)" labels to all sections so the time window context is always visible.	2026-07-10 22:14:56 +00:00
Celes Renata	ed474c521a	fix: update document status after macro classification ... _process_macro_classification never updated document status, leaving macro_event docs stuck in parsed forever. Now marks them extracted on success or extraction_failed on error.	2026-07-10 22:09:02 +00:00
Celes Renata	e8e348d478	fix: update test — empty ticker now filtered out instead of failing extraction	2026-07-10 20:43:46 +00:00
Celes Renata	d9fbd5b660	fix: handle null fields from NuExtract3 in extraction normalization ... - Replace setdefault() with explicit None checks in _normalize_extraction_data() - Coerce null top-level fields (summary, novelty_score, confidence) to defaults - Coerce null company fields (ticker, impact_score, impact_horizon, etc.) to defaults - Filter out company entries with empty ticker after normalization - Prevents schema validation failures when model returns null for required fields	2026-07-10 20:38:50 +00:00
Celes Renata	ca712ad4a0	fix: pipeline health — stuck docs, price fallback, sentiment normalization, signal-engine scale, quality gate ... - Scheduler: lower stale threshold 240→30 min, batch limit 100→500, TTL 14400→3600 - Prediction snapshot: add 24h market_snapshots time-window fallback - Aggregation: add normalize_impact_scores() z-score normalization - Helm: signal-engine replicas → 0 (idle when dual pipeline disabled) - Quality gate: max_snapshot_age_hours 24→48 - Add backfill script for NULL price_at_prediction snapshots - Add PBT bug condition and preservation tests (14 tests)	2026-07-10 20:16:01 +00:00
Celes Renata	a4f51c00e1	fix: beta broker keys in values-beta.yaml (ArgoCD source of truth)	2026-07-09 14:04:59 +00:00
Celes Renata	cf93e2d547	chore: trigger CI rebuild for dashboard fix	2026-07-06 04:06:06 +00:00
Celes Renata	88a6c8f9ee	fix: beta broker keys — use prod Alpaca paper account	2026-07-06 02:46:29 +00:00
Celes Renata	af7b654fb7	fix: vLLM deployment — Recreate strategy, correct service port, single model name ... - Strategy: Recreate (GPU can only be used by one pod at a time) - nuextract-external targetPort: 8080 (nginx proxy port) - served-model-name: numind/NuExtract3 - DB ai_agents updated to model=numind/NuExtract3 provider=vllm	2026-07-04 05:34:04 +00:00
Celes Renata	a322e00659	feat: vLLM serving numind/NuExtract3 on full GPU, ollama disabled ... - Ollama scaled to 0 — vLLM gets entire 16GB GPU - NuExtract3 (9B) running at 80% GPU utilization - Both OLLAMA_BASE_URL and VLLM_BASE_URL point to vLLM service - Fixed gremlin-1 missing default route (no internet access) - Removed GPU resource limit (time-slicing handles allocation) - Removed init container (direct download works with internet fix)	2026-07-03 17:57:19 +00:00
Celes Renata	ecade0dd52	feat: vLLM in K8s via Harbor mirror, pipelines point to internal svc ... - vLLM image mirrored to registry.celestium.life/stonks-oracle/vllm-openai - Deployment uses Harbor image (Docker Hub IPv6 unreachable from cluster) - All 3 pipelines use vllm-external.vllm-service.svc.cluster.local:2701 - K8s manifests at infra/kube-vllm/ and synced to ~/sources/kube/vllm	2026-07-03 17:02:31 +00:00
Celes Renata	3a9894cd03	fix: revert VLLM to bare-metal 192.168.42.254:2701 ... K8s vLLM deployment blocked by Docker Hub IPv6 connectivity issue. Falling back to NixOS systemd vLLM service on esnixi. K8s manifests remain in infra/kube-vllm/ for future use.	2026-07-03 11:40:36 +00:00