Commit Graph
100 Commits
Author SHA1 Message Date
Celes Renata c43865cc9c ci: verify Woodpecker OAuth reconnection
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 21:57:45 +00:00
Celes Renata 987b32d2ff fix: reduce correlation window 30d→7d to prevent OOM (132k→~30k rows)
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 21:46:54 +00:00
Celes Renata e1d1c70d59 ci: trigger pipeline after Woodpecker OAuth fix
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 15:36:17 +00:00
Celes Renata 4f021511fa fix: move correlation matrix to background task — prevents startup timeout/OOM
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 15:23:22 +00:00
Celes Renata 6bde9d2726 fix: increase trading-engine memory limit 512Mi→1024Mi (OOMKilled by correlation matrix)
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 15:20:38 +00:00
Celes Renata 620a8f0d91 fix: add specialist health check, update default namespace to stonks-oracle
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-13 09:17:23 +00:00
Celes Renata f9284ce0a8 docs: update steering files with CI DNS fix, specialist build, and new pitfalls 2026-07-13 09:12:50 +00:00
Celes Renata d4f1b529a5 fix: add missing specialist image build step to CI pipeline 2026-07-13 08:08:23 +00:00
Celes Renata 9617b088bd fix: allow node_modules in Docker context (pre-staged by CI step) 2026-07-13 07:40:19 +00:00
Celes Renata c0d7f2f438 fix: pin registry.npmjs.org IPv4 in /etc/hosts to bypass broken DNS 2026-07-13 07:31:38 +00:00
Celes Renata 7229273461 fix: reduce npm maxsockets to 3, add retries for EAI_AGAIN DNS failures 2026-07-13 07:27:51 +00:00
Celes Renata d18a63aab5 fix: remove invalid npm config, keep http logging for diagnostics 2026-07-13 07:23:40 +00:00
Celes Renata 7df45f7e27 fix: use map format for Woodpecker environment vars 2026-07-13 07:19:47 +00:00
Celes Renata 0d4a812e3d fix: add DNS diagnostics to npm-install-dashboard CI step 2026-07-13 06:58:43 +00:00
Celes Renata 2fff553518 fix: force IPv4 for npm — cluster has no IPv6 egress
Root cause: DNS returns IPv6 (AAAA) records for registry.npmjs.org
first. Alpine musl tries IPv6, which hangs because the cluster has
no IPv6 routing. NODE_OPTIONS=--dns-result-order=ipv4first forces
Node/npm to use IPv4 addresses.
2026-07-13 06:41:46 +00:00
Celes Renata 2d516a725d fix: pre-stage npm ci outside BuildKit sandbox
BuildKit sandbox has no network access for DNS resolution. Instead of
fighting it, run npm ci in a normal Woodpecker step (which has full
network), then COPY the whole frontend dir (including node_modules)
into the Docker build context. The Dockerfile just runs npm run build.
2026-07-13 05:55:41 +00:00
Celes Renata cdad58ae28 fix: add network=host to dashboard buildx build step
Root cause confirmed: BuildKit sandbox creates isolated network namespace
that cannot resolve DNS even though the pod itself can. The buildx_flags
--driver-opt network=host only affects the buildkit daemon creation, not
the actual build steps. Adding network: host passes --network=host to
docker buildx build, making RUN steps use host networking.
2026-07-13 05:52:00 +00:00
Celes Renata 9645d9c261 fix: revert Dockerfile hacks, restart CoreDNS to fix DNS resolution
Root cause: CoreDNS pods were failing to forward external DNS queries
(registry.npmjs.org.celestium.life -> no next plugin). Restarted
CoreDNS deployment. Reverted Dockerfile to original clean state.
2026-07-13 05:20:48 +00:00
Celes Renata 05e6f21a16 fix: set BuildKit worker networkMode=host for dashboard build
The BuildKit sandbox cannot resolve registry.npmjs.org DNS. Setting
networkMode=host in the worker config forces all RUN steps to use
the host network stack, which has confirmed internet access.
2026-07-13 05:02:35 +00:00
Celes Renata d762a2b76f fix: use RUN --network=host for npm ci in BuildKit
BuildKit sandbox DNS cannot resolve registry.npmjs.org. Using
--network=host on the RUN step lets npm use the host network stack
directly, bypassing the broken sandbox DNS.
2026-07-13 04:47:37 +00:00
Celes Renata 46d7699ed0 fix: add explicit DNS resolvers to frontend Dockerfile for BuildKit
BuildKit sandbox does not inherit cluster DNS, causing npm ci to fail
with EAI_AGAIN. Inject Google/Cloudflare DNS into resolv.conf before
running npm ci.
2026-07-13 04:38:59 +00:00
Celes Renata 13729bda98 ci: retry (DNS resolution failure in BuildKit) 2026-07-13 04:19:28 +00:00
Celes Renata b5deeaa0d5 ci: retry build (npm ci hung on previous attempt) 2026-07-13 03:07:01 +00:00
Celes Renata 0aa72cbce1 fix: add numpy and scikit-learn to requirements for v3 calibrators 2026-07-13 02:19:44 +00:00
Celes Renata a72f336ad1 feat: Intelligence Pipeline v3 — full implementation
Multi-stage evidence-grounded inference architecture replacing the
monolithic 9B model extraction pipeline. CPU-first specialist services
handle routine extraction while the 9B vLLM model is preserved for
semantic adjudication of ambiguous cases.

Key components:
- Capability-aware inference gateway (OpenAI-compatible + Ollama)
- Endpoint registry with DB migrations and REST API
- Sentence-aware document segmenter (property tests)
- Deterministic financial parsing with offset integrity
- Symbol resolution with ambiguity detection
- Specialist service (GLiNER2, dynamic batching, K8s deployment)
- Company-specific sentiment (FinBERT, calibration)
- Retrieval-based novelty and duplicate detection
- Confidence calibration pipeline
- Deterministic routing engine (property tests)
- 9B adjudication layer with VRAM gating
- Stock-specific impact model (features, labels, baseline, trained)
- Pipeline orchestrator (state machine, queues, leases, feature flags)
- Bounded parallelism (async workers, semaphore, load shedding)
- Observability (tracing, metrics, alerts)
- Compatibility adapter (v3→v2 golden mapping tests)
- Shadow/canary promotion framework
- Active learning and fine-tuning pipeline

Test results: 1,161 tests pass, ruff lint clean.
All 282 spec tasks completed.
2026-07-13 02:14:59 +00:00
Celes Renata 84634a365e feat: deep pipeline health check script
Checks 12 aspects of the running deployment:
- Pod health (replicas, restarts, OOMKills)
- PostgreSQL connectivity & stats
- Redis connectivity & queue backlog
- MinIO connectivity
- vLLM model availability
- Alpaca broker account status
- Pipeline flow (ingestion→extraction→aggregation→recommendation)
- Model quality gate freshness
- Paper trading portfolio status
- API endpoint health
- Resource usage
- End-to-end latency (doc processing, trend/rec freshness)

Usage: ./scripts/pipeline_health_check.sh [namespace]
2026-07-11 22:35:04 +00:00
Celes Renata 2f966b1caf fix: scheduler memory 128Mi→256Mi, fix backfill script SQL param bug 2026-07-11 17:23:52 +00:00
Celes Renata 81ed2f0803 fix(ci): bypass DNS for clone — use Gitea ClusterIP directly
All pipeline files now specify clone.git.settings.remote with the
Gitea service ClusterIP (10.43.73.77:3000) instead of relying on
DNS resolution of gitea-service.git-server.svc.cluster.local.

This permanently fixes the intermittent DNS resolution failures
that occur when multiple build pipelines clone simultaneously.
2026-07-11 05:36:41 +00:00
Celes Renata 5bd5c521dc ci: dns debug 2026-07-11 05:28:31 +00:00
Celes Renata 9cbb559ac7 ci: retry - coredns permanent fix (dnsPolicy:None + forward 192.168.42.1) 2026-07-11 04:39:28 +00:00
Celes Renata b13232a21f ci: retry after reverting coredns and woodpecker dns to original config 2026-07-11 04:36:20 +00:00
Celes Renata 0cd052a0a7 ci: retry - restored search domains, ndots:2, timeout:10, attempts:5 2026-07-11 04:33:41 +00:00
Celes Renata a2f4398f8d ci: retry - dns config with ndots:3, timeout:5, attempts:5 2026-07-11 04:32:09 +00:00
Celes Renata 76e06d5951 ci: retry after scaling coredns to 4 replicas across all nodes 2026-07-10 23:25:16 +00:00
Celes Renata 96eabf595b ci: retry after adding 192.168.42.1 to woodpecker DNS config 2026-07-10 23:21:18 +00:00
Celes Renata 430554fe0c ci: retry after coredns fix + woodpecker agent restart 2026-07-10 23:18:59 +00:00
Celes Renata 853b85a2af ci: retry after coredns forward fix 2026-07-10 23:14:18 +00:00
Celes Renata 1d104187f4 ci: retry after transient DNS failure 2026-07-10 23:11:32 +00:00
Celes Renata d018c75182 fix: OpsPipeline dashboard — use time-filtered doc stages, add time labels
Document Stages now shows time-filtered counts from /health endpoint
(consistent with Parsing/Extraction/Trends sections) with all-time
totals shown as a subtle subtitle from the SSE stream. Added "(last Xh)"
labels to all sections so the time window context is always visible.
2026-07-10 22:14:56 +00:00
Celes Renata ed474c521a fix: update document status after macro classification
_process_macro_classification never updated document status, leaving
macro_event docs stuck in parsed forever. Now marks them extracted on
success or extraction_failed on error.
2026-07-10 22:09:02 +00:00
Celes Renata e8e348d478 fix: update test — empty ticker now filtered out instead of failing extraction 2026-07-10 20:43:46 +00:00
Celes Renata d9fbd5b660 fix: handle null fields from NuExtract3 in extraction normalization
- Replace setdefault() with explicit None checks in _normalize_extraction_data()
- Coerce null top-level fields (summary, novelty_score, confidence) to defaults
- Coerce null company fields (ticker, impact_score, impact_horizon, etc.) to defaults
- Filter out company entries with empty ticker after normalization
- Prevents schema validation failures when model returns null for required fields
2026-07-10 20:38:50 +00:00
Celes Renata ca712ad4a0 fix: pipeline health — stuck docs, price fallback, sentiment normalization, signal-engine scale, quality gate
- Scheduler: lower stale threshold 240→30 min, batch limit 100→500, TTL 14400→3600
- Prediction snapshot: add 24h market_snapshots time-window fallback
- Aggregation: add normalize_impact_scores() z-score normalization
- Helm: signal-engine replicas → 0 (idle when dual pipeline disabled)
- Quality gate: max_snapshot_age_hours 24→48
- Add backfill script for NULL price_at_prediction snapshots
- Add PBT bug condition and preservation tests (14 tests)
2026-07-10 20:16:01 +00:00
Celes Renata a4f51c00e1 fix: beta broker keys in values-beta.yaml (ArgoCD source of truth) 2026-07-09 14:04:59 +00:00
Celes Renata cf93e2d547 chore: trigger CI rebuild for dashboard fix 2026-07-06 04:06:06 +00:00
Celes Renata 88a6c8f9ee fix: beta broker keys — use prod Alpaca paper account 2026-07-06 02:46:29 +00:00
Celes Renata af7b654fb7 fix: vLLM deployment — Recreate strategy, correct service port, single model name
- Strategy: Recreate (GPU can only be used by one pod at a time)
- nuextract-external targetPort: 8080 (nginx proxy port)
- served-model-name: numind/NuExtract3
- DB ai_agents updated to model=numind/NuExtract3 provider=vllm
2026-07-04 05:34:04 +00:00
Celes Renata a322e00659 feat: vLLM serving numind/NuExtract3 on full GPU, ollama disabled
- Ollama scaled to 0 — vLLM gets entire 16GB GPU
- NuExtract3 (9B) running at 80% GPU utilization
- Both OLLAMA_BASE_URL and VLLM_BASE_URL point to vLLM service
- Fixed gremlin-1 missing default route (no internet access)
- Removed GPU resource limit (time-slicing handles allocation)
- Removed init container (direct download works with internet fix)
2026-07-03 17:57:19 +00:00
Celes Renata ecade0dd52 feat: vLLM in K8s via Harbor mirror, pipelines point to internal svc
- vLLM image mirrored to registry.celestium.life/stonks-oracle/vllm-openai
- Deployment uses Harbor image (Docker Hub IPv6 unreachable from cluster)
- All 3 pipelines use vllm-external.vllm-service.svc.cluster.local:2701
- K8s manifests at infra/kube-vllm/ and synced to ~/sources/kube/vllm
2026-07-03 17:02:31 +00:00
Celes Renata 3a9894cd03 fix: revert VLLM to bare-metal 192.168.42.254:2701
K8s vLLM deployment blocked by Docker Hub IPv6 connectivity issue.
Falling back to NixOS systemd vLLM service on esnixi.
K8s manifests remain in infra/kube-vllm/ for future use.
2026-07-03 11:40:36 +00:00
Celes Renata 4ac10ab4f6 fix: point VLLM_BASE_URL to K8s internal vllm-service
vLLM now runs in vllm-service namespace with metrics proxy,
accessible at vllm-external.vllm-service.svc.cluster.local:2701
2026-07-03 11:38:24 +00:00
Celes Renata 3996ec6ef5 feat: switch models — ollama qwen3.5:4b-fast, vllm numind/NuExtract3
- Ollama: qwen3.5:4b-fast with NUM_PARALLEL=8, num_ctx=16384 (~6GB VRAM)
- vLLM: numind/NuExtract3 at 192.168.42.254:2701, gpu-memory-util=0.45
- All 3 pipelines (beta/paper/live) updated with correct endpoints
- Ollama proxy at 10.1.1.12:2701 (K8s ollama-metrics)
- vLLM proxy at 192.168.42.254:2701 (NixOS nginx)
2026-07-03 11:15:28 +00:00
Celes Renata d1533ed31d chore: test CI after CoreDNS restart 2026-07-03 10:17:38 +00:00
Celes Renata 02eb8e619f chore: trigger CI rebuild 2026-07-03 09:55:37 +00:00
Celes Renata a6aceebe18 fix: dashboards limit exceeds API max (500→200)
Sentiment Heatmap and Prediction Accuracy passed limit=500 but
the trends and recommendations endpoints enforce limit<=200,
causing a 422 validation error that made both panels show blank.
2026-07-03 09:35:55 +00:00
Celes Renata 14a9b4fcc1 fix: ops pipeline — rate limit, validation cycle, snapshots, config query, rejection reason
- Polygon rate limit env-configurable, default 5 (free tier)
- v3_engine_enabled reads from JSONB config column correctly
- Validation cycle (outcome eval + metrics) wired into scheduler hourly
- Daily portfolio/risk snapshots after 16:30 ET with idempotency
- Prediction snapshot price fallback from positions table
- Order rejection_reason + rejected_at persisted on insert
- Lake-publisher scaled to 0 replicas (redundant)
- Test fix: isolate per-type rate limit tests from Polygon global
2026-07-03 08:29:24 +00:00
Celes Renata b70304ad6c fix: correct Alpaca paper trading credentials for beta 2026-06-30 15:24:30 +00:00
Celes Renata fdb54b77ce fix: update Redis password, Ollama URL, and broker keys for new cluster 2026-06-28 22:03:26 +00:00
Celes Renata 106b07c0f0 fix: ruff lint fixes for v3 test files 2026-06-27 12:32:52 +00:00
Celes Renata b4bf0f2361 feat: math core v3 engine upgrade 2026-06-27 12:21:41 +00:00
Celes Renata 5ed9f001b1 docs: add bash exclamation mark pitfall to steering file 2026-05-02 08:33:40 +00:00
Celes Renata 18f857e1a3 feat: add live-math stage with dedicated paper trading account
New ArgoCD stage 'stonks-live-math' running the dual-pipeline signal
engine against a separate Alpaca paper account for production validation.

- Namespace: stonks-live-math
- Database: stonks_live_math (Redis DB 3)
- Broker: PK64RS7NH24XPBI3IDEU3BB72Y (paper-api.alpaca.markets)
- Ingress: stonks-math-*.celestium.life
- Deploys alongside existing live stage (same image tag)
2026-05-02 08:19:47 +00:00
Celes Renata 61bcb5aa57 ci: add signal-engine image build to Woodpecker pipeline 2026-05-02 07:43:59 +00:00
Celes Renata f468e30af0 feat: implement dual-pipeline signal engine service
New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.

Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
  gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading

Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant

Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)
2026-05-02 07:32:26 +00:00
Celes Renata 7e2343ec2c feat: add Reports page and detail view to frontend
- Reports list page with type filter (daily/weekly)
- Report detail page with all sections: P&L, recommendation accuracy,
  position performance table, risk metrics, model quality windows
- Executive summary card, validation warnings display
- Nav item under Trading group
- Routes: /reports and /reports/:id
2026-05-02 00:27:34 +00:00
Celes Renata 3a8c6f6c80 fix: add CACHE_BUST ARG to Dockerfile + build args to force fresh COPY layers per commit 2026-05-01 23:23:49 +00:00
Celes Renata 139fbd6342 fix: add no_cache to all Woodpecker service builds to prevent stale COPY layers 2026-05-01 23:11:21 +00:00
Celes Renata 63b62c5e9f chore: RUN echo cache-bust to force buildx layer invalidation 2026-05-01 22:57:29 +00:00
Celes Renata 3acfdfd11f chore: move cache-bust before COPY to invalidate buildx layer cache 2026-05-01 22:48:12 +00:00
Celes Renata ab212db8be chore: bust Docker buildx cache for services/ layer 2026-05-01 22:39:38 +00:00
Celes Renata 7102358f51 chore: retrigger CI (bust Docker buildx cache for services/ layer) 2026-05-01 22:32:23 +00:00
Celes Renata bc077bfcc8 feat: trading feedback engine — periodic performance reports with AI summarization
- Migration 038: trading_reports table + report-summarizer agent seed
- 6 reporting modules: models, collector, sections, validator, summarizer, generator
- API endpoints: GET /api/reports (paginated, filterable), GET /api/reports/{id}
- Frontend hooks: useReports, useReport with TanStack Query
- Scheduler: daily (after 16:30 ET) and weekly (Saturday) report triggers
- Redis queue consumer for async report generation with retry/dedup
- 5 property-based tests (chunking, serialization, validation, accuracy, deltas)
- 109 unit/integration tests across all modules
- 6 frontend hook tests with MSW mocks
2026-05-01 22:13:09 +00:00
Celes Renata 376fcb4bb4 feat: add 12 integrity check saved queries for SQL Explorer (migration 037)
Prefixed with ⚕ for easy identification. Checks: duplicate snapshots,
orphaned links, evidence count mismatches, contribution score sums,
canonical key consistency, out-of-range values, unmatched snapshots,
zero evidence rate, duplicate count mismatches, missing prices,
outcome integrity, and pipeline health summary.
2026-05-01 19:20:56 +00:00
Celes Renata cc21fd9e8f feat: add 15 validation saved queries for SQL Explorer (migration 036)
Prediction overview, confidence distribution, evidence dedup quality,
source/catalyst breakdown, win rate by ticker/horizon, model quality
timeline, quality gate status, high-duplicate predictions, excess vs SPY.
2026-05-01 19:15:54 +00:00
Celes Renata affb65d7f4 fix: point VLLM_BASE_URL to Ollama (10.1.1.12:2701) and update model name
vLLM service on 192.168.42.254:8000 was down causing extraction failures.
Also updated ai_agents table to use ollama provider directly.
2026-05-01 18:56:55 +00:00
Celes Renata 32535540fe feat: add no-premature-simplification rule to steering docs 2026-05-01 04:32:05 +00:00
Celes Renata 751cce0509 feat: add documentation maintenance rules to steering, update migration number to 036 2026-05-01 04:29:52 +00:00
Celes Renata 0732894414 fix: widen horizon/mode columns in prediction_snapshots to VARCHAR(50)
time_horizon values like 'position_30d_90d (proj:30d)' exceed VARCHAR(20)
2026-05-01 04:27:19 +00:00
Celes Renata d9110d03a6 fix: use di.document_id instead of dir.document_id in evidence query
document_impact_records has intelligence_id, not document_id.
The document_id lives on document_intelligence.
2026-05-01 04:06:55 +00:00
Celes Renata 76f6bd5677 fix: quote reserved keyword 'window' in prediction_snapshots SQL 2026-05-01 03:40:48 +00:00
Celes Renata 32d290bea7 fix: ruff lint — sort imports, remove unused GateThresholdResult import 2026-05-01 03:08:21 +00:00
Celes Renata 7fcc8a6c07 feat: model validation, calibration, and signal quality layer
- Migration 035: prediction_snapshots, prediction_outcomes, signal_evidence_links, model_metric_snapshots tables + SQL views
- Prediction snapshot writer with canonical evidence keys, duplicate detection, contribution scores
- Outcome evaluator across 5 horizons (1h, 6h, 1d, 7d, 30d)
- Metrics engine: ECE, Brier score, IC, Rank IC, benchmark comparison
- Attribution engine: per-source, per-catalyst, per-layer performance
- Calibration engine: Bayesian shrinkage source reliability
- Quality gate for live trading eligibility with configurable thresholds
- 7 new /api/validation/* endpoints
- Upgraded OpsModel dashboard with validation tab
- Enhanced recommendation display with calibration context
- Backtest replay validation mode
- 86 Python tests (unit + property-based), 179 frontend tests passing
2026-05-01 03:04:58 +00:00
Celes Renata 5d2ffd9163 fix: vertical market markers need yAxisId + move legend below date labels
- Add yAxisId='left' to vertical ReferenceLine (required by Recharts)
- Increase bottom margin from 40 to 70 for date label + legend space
- Legend uses verticalAlign='bottom' with 24px top padding
2026-04-30 23:31:55 +00:00
Celes Renata 6169efdc89 fix: copy scripts/ into Docker image + graceful backfill fallback
- Dockerfile now copies scripts/ directory into /app/scripts/
- Init container uses sh -c with fallback so missing script
  doesn't crash the pod (graceful degradation until image rebuilds)
2026-04-30 23:18:43 +00:00
Celes Renata c42f2223d8 fix: move backfill logic to standalone script (fixes YAML parse error)
Inline Python with f-strings and colons broke Helm YAML parsing.
Moved to scripts/backfill_market_data.py and call it directly
from the init container command.
2026-04-30 23:01:20 +00:00
Celes Renata 1f08820f11 feat: fullscreen chart expand + market open/close vertical markers
- Click 'Expand' button to view the trend chart fullscreen
- Press Escape or click outside to close
- Fullscreen uses full viewport height for better readability
- Green dashed vertical lines at 9:30 AM ET (market open)
- Red dashed vertical lines at 4:00 PM ET (market close)
- Markers only shown on intraday and 1d windows (hidden on 7d+)
- Chart content extracted into shared function to avoid duplication
2026-04-30 22:44:00 +00:00
Celes Renata 2f2ea65fb4 fix: keep Alpaca price as primary (includes after-hours), add polygon_price field
Alpaca's current_price reflects extended hours trading which is more
current than Polygon's regular session close. Keep it as the display
price. Add polygon_price as a reference field in the API response.
2026-04-30 22:35:22 +00:00
Celes Renata 9a60ce127b fix: overlay Polygon prices on positions instead of stale Alpaca prices
Alpaca paper trading returns inaccurate current_price values.
The positions endpoint now uses the latest Polygon close from
market_snapshots and recomputes unrealized P&L from that.
2026-04-30 22:32:21 +00:00
Celes Renata 414f476620 fix: auto-backfill market data on startup + fix trading dedup bug
1. Add backfill-market-data init container to scheduler deployment.
   On vanilla start (< 50 bars in market_snapshots), fetches 90 days
   of daily OHLCV from Polygon directly via asyncpg + httpx.

2. Fix trading engine dedup key placement. Previously the Redis dedup
   key was set BEFORE evaluate_recommendation(), so recs skipped as
   outside_trading_window were permanently deduped and never retried
   when the market opened. Now the dedup key is only set AFTER the
   decision for non-retryable outcomes (act, confidence_too_low, etc).
   outside_trading_window skips are retryable.
2026-04-30 14:08:54 +00:00
Celes Renata a5f2bcde55 fix: set MARKET_DATA_BASE_URL to polygon in base values.yaml 2026-04-30 07:20:13 +00:00
Celes Renata 34ffdad00c fix: wait for scheduler migrations after factory reset scale-up
After scale_up, restart the scheduler and wait for it to be ready
before other services start. The scheduler's init containers run
migrations, so this ensures tables exist before ingestion/extractor
pods try to query them.
2026-04-30 07:01:01 +00:00
Celes Renata 601b85764b fix: mount stonks-market-secrets on query-api for backfill endpoint 2026-04-30 06:33:26 +00:00
Celes Renata 51b6f3d34a ci: trigger rebuild 2026-04-30 06:24:05 +00:00
Celes Renata bddaf44ffc feat: 90-day price range Y-axis scaling with breakthrough annotations
Backend:
- GET /api/market/prices/{ticker} now returns { bars, range_90d }
  with 90-day low/high computed from market_snapshots
- POST /api/market/backfill/{ticker} fetches 90 days of daily bars
  from Polygon and inserts missing bars into market_snapshots
- POST /api/market/backfill-all does the same for all active tickers

Frontend:
- Right Y-axis domain scaled to 90-day min/max (with 3% padding)
- Green dashed reference line at 90-day high
- Red dashed reference line at 90-day low
- Labels show exact price on each reference line
- Default limit bumped to 200 bars
2026-04-30 06:09:05 +00:00
Celes Renata 5209cc522e feat: enhanced paper trading reset with capital and reserve controls
- Add initial capital input (toggle between broker balance or custom amount)
- Add reserve/active pool split slider (0-50%, default 20%)
- Backend accepts reserve_pct in reset request body
- Note in UI that Alpaca balance reset requires Alpaca dashboard
- Confirmation dialog shows exact capital and split being applied
2026-04-30 05:49:38 +00:00
Celes Renata fa18b1a7c2 fix: populate broker and market API secrets in base values.yaml
ArgoCD was syncing empty secrets back over manual patches.
Credentials now match stonks-paper (same Alpaca paper + Polygon keys).
2026-04-30 05:05:43 +00:00
Celes Renata 4b254adad2 fix: scale ingestion and extractor to 1 replica in prod 2026-04-30 05:04:20 +00:00
Celes Renata af2e554edd fix: switch ai_agents seed from vllm to ollama qwen3.5:9b-fast
Live DB already updated via direct SQL. This ensures future
DB rebuilds and new deployments use ollama as the default provider.
2026-04-30 04:46:04 +00:00
Celes Renata facce5dbb5 fix: add TTL and backoffLimit to market-hours-check jobs
Prevents accumulation of completed/failed verification jobs.
- ttlSecondsAfterFinished: 3600 (auto-cleanup after 1h)
- backoffLimit: 0 (no retries on intentional market-hours failure)
2026-04-30 04:33:50 +00:00
Celes Renata 2f2c0d24f6 fix: remove unused columns constant (TS6133) 2026-04-30 03:21:32 +00:00