admin/stonks-oracle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
564 Commits 1 Branch 1 Tag
d762a2b76f2d8b6c35ed57d41f734c9f0cec9606
Commit Graph
564 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Celes Renata d762a2b76f fix: use RUN --network=host for npm ci in BuildKit 
BuildKit sandbox DNS cannot resolve registry.npmjs.org. Using
--network=host on the RUN step lets npm use the host network stack
directly, bypassing the broken sandbox DNS.
 2026-07-13 04:47:37 +00:00
Celes Renata 46d7699ed0 fix: add explicit DNS resolvers to frontend Dockerfile for BuildKit 
BuildKit sandbox does not inherit cluster DNS, causing npm ci to fail
with EAI_AGAIN. Inject Google/Cloudflare DNS into resolv.conf before
running npm ci.
 2026-07-13 04:38:59 +00:00
Celes Renata 13729bda98 ci: retry (DNS resolution failure in BuildKit) 2026-07-13 04:19:28 +00:00
Celes Renata b5deeaa0d5 ci: retry build (npm ci hung on previous attempt) 2026-07-13 03:07:01 +00:00
Celes Renata 0aa72cbce1 fix: add numpy and scikit-learn to requirements for v3 calibrators 2026-07-13 02:19:44 +00:00
Celes Renata a72f336ad1 feat: Intelligence Pipeline v3 — full implementation 
Multi-stage evidence-grounded inference architecture replacing the
monolithic 9B model extraction pipeline. CPU-first specialist services
handle routine extraction while the 9B vLLM model is preserved for
semantic adjudication of ambiguous cases.

Key components:
- Capability-aware inference gateway (OpenAI-compatible + Ollama)
- Endpoint registry with DB migrations and REST API
- Sentence-aware document segmenter (property tests)
- Deterministic financial parsing with offset integrity
- Symbol resolution with ambiguity detection
- Specialist service (GLiNER2, dynamic batching, K8s deployment)
- Company-specific sentiment (FinBERT, calibration)
- Retrieval-based novelty and duplicate detection
- Confidence calibration pipeline
- Deterministic routing engine (property tests)
- 9B adjudication layer with VRAM gating
- Stock-specific impact model (features, labels, baseline, trained)
- Pipeline orchestrator (state machine, queues, leases, feature flags)
- Bounded parallelism (async workers, semaphore, load shedding)
- Observability (tracing, metrics, alerts)
- Compatibility adapter (v3→v2 golden mapping tests)
- Shadow/canary promotion framework
- Active learning and fine-tuning pipeline

Test results: 1,161 tests pass, ruff lint clean.
All 282 spec tasks completed.
 2026-07-13 02:14:59 +00:00
Celes Renata 84634a365e feat: deep pipeline health check script 
Checks 12 aspects of the running deployment:
- Pod health (replicas, restarts, OOMKills)
- PostgreSQL connectivity & stats
- Redis connectivity & queue backlog
- MinIO connectivity
- vLLM model availability
- Alpaca broker account status
- Pipeline flow (ingestion→extraction→aggregation→recommendation)
- Model quality gate freshness
- Paper trading portfolio status
- API endpoint health
- Resource usage
- End-to-end latency (doc processing, trend/rec freshness)

Usage: ./scripts/pipeline_health_check.sh [namespace]
 2026-07-11 22:35:04 +00:00
Celes Renata 2f966b1caf fix: scheduler memory 128Mi→256Mi, fix backfill script SQL param bug 2026-07-11 17:23:52 +00:00
Celes Renata 81ed2f0803 fix(ci): bypass DNS for clone — use Gitea ClusterIP directly 
All pipeline files now specify clone.git.settings.remote with the
Gitea service ClusterIP (10.43.73.77:3000) instead of relying on
DNS resolution of gitea-service.git-server.svc.cluster.local.

This permanently fixes the intermittent DNS resolution failures
that occur when multiple build pipelines clone simultaneously.
 2026-07-11 05:36:41 +00:00
Celes Renata 5bd5c521dc ci: dns debug 2026-07-11 05:28:31 +00:00
Celes Renata 9cbb559ac7 ci: retry - coredns permanent fix (dnsPolicy:None + forward 192.168.42.1) 2026-07-11 04:39:28 +00:00
Celes Renata b13232a21f ci: retry after reverting coredns and woodpecker dns to original config 2026-07-11 04:36:20 +00:00
Celes Renata 0cd052a0a7 ci: retry - restored search domains, ndots:2, timeout:10, attempts:5 2026-07-11 04:33:41 +00:00
Celes Renata a2f4398f8d ci: retry - dns config with ndots:3, timeout:5, attempts:5 2026-07-11 04:32:09 +00:00
Celes Renata 76e06d5951 ci: retry after scaling coredns to 4 replicas across all nodes 2026-07-10 23:25:16 +00:00
Celes Renata 96eabf595b ci: retry after adding 192.168.42.1 to woodpecker DNS config 2026-07-10 23:21:18 +00:00
Celes Renata 430554fe0c ci: retry after coredns fix + woodpecker agent restart 2026-07-10 23:18:59 +00:00
Celes Renata 853b85a2af ci: retry after coredns forward fix 2026-07-10 23:14:18 +00:00
Celes Renata 1d104187f4 ci: retry after transient DNS failure 2026-07-10 23:11:32 +00:00
Celes Renata d018c75182 fix: OpsPipeline dashboard — use time-filtered doc stages, add time labels 
Document Stages now shows time-filtered counts from /health endpoint
(consistent with Parsing/Extraction/Trends sections) with all-time
totals shown as a subtle subtitle from the SSE stream. Added "(last Xh)"
labels to all sections so the time window context is always visible.
 2026-07-10 22:14:56 +00:00
Celes Renata ed474c521a fix: update document status after macro classification 
_process_macro_classification never updated document status, leaving
macro_event docs stuck in parsed forever. Now marks them extracted on
success or extraction_failed on error.
 2026-07-10 22:09:02 +00:00
Celes Renata e8e348d478 fix: update test — empty ticker now filtered out instead of failing extraction 2026-07-10 20:43:46 +00:00
Celes Renata d9fbd5b660 fix: handle null fields from NuExtract3 in extraction normalization 
- Replace setdefault() with explicit None checks in _normalize_extraction_data()
- Coerce null top-level fields (summary, novelty_score, confidence) to defaults
- Coerce null company fields (ticker, impact_score, impact_horizon, etc.) to defaults
- Filter out company entries with empty ticker after normalization
- Prevents schema validation failures when model returns null for required fields
 2026-07-10 20:38:50 +00:00
Celes Renata ca712ad4a0 fix: pipeline health — stuck docs, price fallback, sentiment normalization, signal-engine scale, quality gate 
- Scheduler: lower stale threshold 240→30 min, batch limit 100→500, TTL 14400→3600
- Prediction snapshot: add 24h market_snapshots time-window fallback
- Aggregation: add normalize_impact_scores() z-score normalization
- Helm: signal-engine replicas → 0 (idle when dual pipeline disabled)
- Quality gate: max_snapshot_age_hours 24→48
- Add backfill script for NULL price_at_prediction snapshots
- Add PBT bug condition and preservation tests (14 tests)
 2026-07-10 20:16:01 +00:00
Celes Renata a4f51c00e1 fix: beta broker keys in values-beta.yaml (ArgoCD source of truth) 2026-07-09 14:04:59 +00:00
Celes Renata cf93e2d547 chore: trigger CI rebuild for dashboard fix 2026-07-06 04:06:06 +00:00
Celes Renata 88a6c8f9ee fix: beta broker keys — use prod Alpaca paper account 2026-07-06 02:46:29 +00:00
Celes Renata af7b654fb7 fix: vLLM deployment — Recreate strategy, correct service port, single model name 
- Strategy: Recreate (GPU can only be used by one pod at a time)
- nuextract-external targetPort: 8080 (nginx proxy port)
- served-model-name: numind/NuExtract3
- DB ai_agents updated to model=numind/NuExtract3 provider=vllm
 2026-07-04 05:34:04 +00:00
Celes Renata a322e00659 feat: vLLM serving numind/NuExtract3 on full GPU, ollama disabled 
- Ollama scaled to 0 — vLLM gets entire 16GB GPU
- NuExtract3 (9B) running at 80% GPU utilization
- Both OLLAMA_BASE_URL and VLLM_BASE_URL point to vLLM service
- Fixed gremlin-1 missing default route (no internet access)
- Removed GPU resource limit (time-slicing handles allocation)
- Removed init container (direct download works with internet fix)
 2026-07-03 17:57:19 +00:00
Celes Renata ecade0dd52 feat: vLLM in K8s via Harbor mirror, pipelines point to internal svc 
- vLLM image mirrored to registry.celestium.life/stonks-oracle/vllm-openai
- Deployment uses Harbor image (Docker Hub IPv6 unreachable from cluster)
- All 3 pipelines use vllm-external.vllm-service.svc.cluster.local:2701
- K8s manifests at infra/kube-vllm/ and synced to ~/sources/kube/vllm
 2026-07-03 17:02:31 +00:00
Celes Renata 3a9894cd03 fix: revert VLLM to bare-metal 192.168.42.254:2701 
K8s vLLM deployment blocked by Docker Hub IPv6 connectivity issue.
Falling back to NixOS systemd vLLM service on esnixi.
K8s manifests remain in infra/kube-vllm/ for future use.
 2026-07-03 11:40:36 +00:00
Celes Renata 4ac10ab4f6 fix: point VLLM_BASE_URL to K8s internal vllm-service 
vLLM now runs in vllm-service namespace with metrics proxy,
accessible at vllm-external.vllm-service.svc.cluster.local:2701
 2026-07-03 11:38:24 +00:00
Celes Renata 3996ec6ef5 feat: switch models — ollama qwen3.5:4b-fast, vllm numind/NuExtract3 
- Ollama: qwen3.5:4b-fast with NUM_PARALLEL=8, num_ctx=16384 (~6GB VRAM)
- vLLM: numind/NuExtract3 at 192.168.42.254:2701, gpu-memory-util=0.45
- All 3 pipelines (beta/paper/live) updated with correct endpoints
- Ollama proxy at 10.1.1.12:2701 (K8s ollama-metrics)
- vLLM proxy at 192.168.42.254:2701 (NixOS nginx)
 2026-07-03 11:15:28 +00:00
Celes Renata d1533ed31d chore: test CI after CoreDNS restart 2026-07-03 10:17:38 +00:00
Celes Renata 02eb8e619f chore: trigger CI rebuild 2026-07-03 09:55:37 +00:00
Celes Renata a6aceebe18 fix: dashboards limit exceeds API max (500→200) 
Sentiment Heatmap and Prediction Accuracy passed limit=500 but
the trends and recommendations endpoints enforce limit<=200,
causing a 422 validation error that made both panels show blank.
 2026-07-03 09:35:55 +00:00
Celes Renata 14a9b4fcc1 fix: ops pipeline — rate limit, validation cycle, snapshots, config query, rejection reason 
- Polygon rate limit env-configurable, default 5 (free tier)
- v3_engine_enabled reads from JSONB config column correctly
- Validation cycle (outcome eval + metrics) wired into scheduler hourly
- Daily portfolio/risk snapshots after 16:30 ET with idempotency
- Prediction snapshot price fallback from positions table
- Order rejection_reason + rejected_at persisted on insert
- Lake-publisher scaled to 0 replicas (redundant)
- Test fix: isolate per-type rate limit tests from Polygon global
 2026-07-03 08:29:24 +00:00
Celes Renata b70304ad6c fix: correct Alpaca paper trading credentials for beta 2026-06-30 15:24:30 +00:00
Celes Renata fdb54b77ce fix: update Redis password, Ollama URL, and broker keys for new cluster 2026-06-28 22:03:26 +00:00
Celes Renata 106b07c0f0 fix: ruff lint fixes for v3 test files 2026-06-27 12:32:52 +00:00
Celes Hillyerd 5d7aaacc9d ci: trigger full image build for beta deployment 2026-06-27 05:28:15 -07:00
Celes Renata b4bf0f2361 feat: math core v3 engine upgrade 2026-06-27 12:21:41 +00:00
Celes Hillyerd 365bc5d4b7 ci: harbor projects created (dockerhub-cache, ghcr-cache, stonks-oracle) 2026-06-26 03:38:40 -07:00
Celes Hillyerd cbd0e156e7 ci: LE cert issued for registry.celestium.life 2026-06-26 03:35:41 -07:00
Celes Hillyerd bf9b7e65dd revert: restore registry cache, investigate LE cert 2026-06-26 03:28:59 -07:00
Celes Hillyerd b516d6805f ci: fix remaining dockerhub-cache references in test/finalize 2026-06-26 03:27:25 -07:00
Celes Hillyerd a0eaa7d6c8 ci: pull from Docker Hub directly, remove broken registry cache/mirrors 2026-06-26 03:27:12 -07:00
Celes Hillyerd 17a044de0f ci: fix buildkit_config setting name (was buildkitd_config) 2026-06-26 03:21:46 -07:00
Celes Hillyerd cf38e6aabc ci: fix buildkitd_config YAML format for insecure registry 2026-06-26 03:16:00 -07:00
Celes Hillyerd 1ef016e652 ci: webhook fixed to internal URL 2026-06-26 03:11:05 -07:00