Celes Renata
9cbb559ac7
ci: retry - coredns permanent fix (dnsPolicy:None + forward 192.168.42.1)
2026-07-11 04:39:28 +00:00
Celes Renata
b13232a21f
ci: retry after reverting coredns and woodpecker dns to original config
2026-07-11 04:36:20 +00:00
Celes Renata
0cd052a0a7
ci: retry - restored search domains, ndots:2, timeout:10, attempts:5
2026-07-11 04:33:41 +00:00
Celes Renata
a2f4398f8d
ci: retry - dns config with ndots:3, timeout:5, attempts:5
2026-07-11 04:32:09 +00:00
Celes Renata
76e06d5951
ci: retry after scaling coredns to 4 replicas across all nodes
2026-07-10 23:25:16 +00:00
Celes Renata
96eabf595b
ci: retry after adding 192.168.42.1 to woodpecker DNS config
2026-07-10 23:21:18 +00:00
Celes Renata
430554fe0c
ci: retry after coredns fix + woodpecker agent restart
2026-07-10 23:18:59 +00:00
Celes Renata
853b85a2af
ci: retry after coredns forward fix
2026-07-10 23:14:18 +00:00
Celes Renata
1d104187f4
ci: retry after transient DNS failure
2026-07-10 23:11:32 +00:00
Celes Renata
d018c75182
fix: OpsPipeline dashboard — use time-filtered doc stages, add time labels
...
Document Stages now shows time-filtered counts from /health endpoint
(consistent with Parsing/Extraction/Trends sections) with all-time
totals shown as a subtle subtitle from the SSE stream. Added "(last Xh)"
labels to all sections so the time window context is always visible.
2026-07-10 22:14:56 +00:00
Celes Renata
ed474c521a
fix: update document status after macro classification
...
_process_macro_classification never updated document status, leaving
macro_event docs stuck in parsed forever. Now marks them extracted on
success or extraction_failed on error.
2026-07-10 22:09:02 +00:00
Celes Renata
e8e348d478
fix: update test — empty ticker now filtered out instead of failing extraction
2026-07-10 20:43:46 +00:00
Celes Renata
d9fbd5b660
fix: handle null fields from NuExtract3 in extraction normalization
...
- Replace setdefault() with explicit None checks in _normalize_extraction_data()
- Coerce null top-level fields (summary, novelty_score, confidence) to defaults
- Coerce null company fields (ticker, impact_score, impact_horizon, etc.) to defaults
- Filter out company entries with empty ticker after normalization
- Prevents schema validation failures when model returns null for required fields
2026-07-10 20:38:50 +00:00
Celes Renata
ca712ad4a0
fix: pipeline health — stuck docs, price fallback, sentiment normalization, signal-engine scale, quality gate
...
- Scheduler: lower stale threshold 240→30 min, batch limit 100→500, TTL 14400→3600
- Prediction snapshot: add 24h market_snapshots time-window fallback
- Aggregation: add normalize_impact_scores() z-score normalization
- Helm: signal-engine replicas → 0 (idle when dual pipeline disabled)
- Quality gate: max_snapshot_age_hours 24→48
- Add backfill script for NULL price_at_prediction snapshots
- Add PBT bug condition and preservation tests (14 tests)
2026-07-10 20:16:01 +00:00
Celes Renata
a4f51c00e1
fix: beta broker keys in values-beta.yaml (ArgoCD source of truth)
2026-07-09 14:04:59 +00:00
Celes Renata
cf93e2d547
chore: trigger CI rebuild for dashboard fix
2026-07-06 04:06:06 +00:00
Celes Renata
88a6c8f9ee
fix: beta broker keys — use prod Alpaca paper account
2026-07-06 02:46:29 +00:00
Celes Renata
af7b654fb7
fix: vLLM deployment — Recreate strategy, correct service port, single model name
...
- Strategy: Recreate (GPU can only be used by one pod at a time)
- nuextract-external targetPort: 8080 (nginx proxy port)
- served-model-name: numind/NuExtract3
- DB ai_agents updated to model=numind/NuExtract3 provider=vllm
2026-07-04 05:34:04 +00:00
Celes Renata
a322e00659
feat: vLLM serving numind/NuExtract3 on full GPU, ollama disabled
...
- Ollama scaled to 0 — vLLM gets entire 16GB GPU
- NuExtract3 (9B) running at 80% GPU utilization
- Both OLLAMA_BASE_URL and VLLM_BASE_URL point to vLLM service
- Fixed gremlin-1 missing default route (no internet access)
- Removed GPU resource limit (time-slicing handles allocation)
- Removed init container (direct download works with internet fix)
2026-07-03 17:57:19 +00:00
Celes Renata
ecade0dd52
feat: vLLM in K8s via Harbor mirror, pipelines point to internal svc
...
- vLLM image mirrored to registry.celestium.life/stonks-oracle/vllm-openai
- Deployment uses Harbor image (Docker Hub IPv6 unreachable from cluster)
- All 3 pipelines use vllm-external.vllm-service.svc.cluster.local:2701
- K8s manifests at infra/kube-vllm/ and synced to ~/sources/kube/vllm
2026-07-03 17:02:31 +00:00
Celes Renata
3a9894cd03
fix: revert VLLM to bare-metal 192.168.42.254:2701
...
K8s vLLM deployment blocked by Docker Hub IPv6 connectivity issue.
Falling back to NixOS systemd vLLM service on esnixi.
K8s manifests remain in infra/kube-vllm/ for future use.
2026-07-03 11:40:36 +00:00
Celes Renata
4ac10ab4f6
fix: point VLLM_BASE_URL to K8s internal vllm-service
...
vLLM now runs in vllm-service namespace with metrics proxy,
accessible at vllm-external.vllm-service.svc.cluster.local:2701
2026-07-03 11:38:24 +00:00
Celes Renata
3996ec6ef5
feat: switch models — ollama qwen3.5:4b-fast, vllm numind/NuExtract3
...
- Ollama: qwen3.5:4b-fast with NUM_PARALLEL=8, num_ctx=16384 (~6GB VRAM)
- vLLM: numind/NuExtract3 at 192.168.42.254:2701, gpu-memory-util=0.45
- All 3 pipelines (beta/paper/live) updated with correct endpoints
- Ollama proxy at 10.1.1.12:2701 (K8s ollama-metrics)
- vLLM proxy at 192.168.42.254:2701 (NixOS nginx)
2026-07-03 11:15:28 +00:00
Celes Renata
d1533ed31d
chore: test CI after CoreDNS restart
2026-07-03 10:17:38 +00:00
Celes Renata
02eb8e619f
chore: trigger CI rebuild
2026-07-03 09:55:37 +00:00
Celes Renata
a6aceebe18
fix: dashboards limit exceeds API max (500→200)
...
Sentiment Heatmap and Prediction Accuracy passed limit=500 but
the trends and recommendations endpoints enforce limit<=200,
causing a 422 validation error that made both panels show blank.
2026-07-03 09:35:55 +00:00
Celes Renata
14a9b4fcc1
fix: ops pipeline — rate limit, validation cycle, snapshots, config query, rejection reason
...
- Polygon rate limit env-configurable, default 5 (free tier)
- v3_engine_enabled reads from JSONB config column correctly
- Validation cycle (outcome eval + metrics) wired into scheduler hourly
- Daily portfolio/risk snapshots after 16:30 ET with idempotency
- Prediction snapshot price fallback from positions table
- Order rejection_reason + rejected_at persisted on insert
- Lake-publisher scaled to 0 replicas (redundant)
- Test fix: isolate per-type rate limit tests from Polygon global
2026-07-03 08:29:24 +00:00
Celes Renata
b70304ad6c
fix: correct Alpaca paper trading credentials for beta
2026-06-30 15:24:30 +00:00
Celes Renata
fdb54b77ce
fix: update Redis password, Ollama URL, and broker keys for new cluster
2026-06-28 22:03:26 +00:00
Celes Renata
106b07c0f0
fix: ruff lint fixes for v3 test files
2026-06-27 12:32:52 +00:00
Celes Hillyerd
5d7aaacc9d
ci: trigger full image build for beta deployment
2026-06-27 05:28:15 -07:00
Celes Renata
b4bf0f2361
feat: math core v3 engine upgrade
2026-06-27 12:21:41 +00:00
Celes Hillyerd
365bc5d4b7
ci: harbor projects created (dockerhub-cache, ghcr-cache, stonks-oracle)
2026-06-26 03:38:40 -07:00
Celes Hillyerd
cbd0e156e7
ci: LE cert issued for registry.celestium.life
2026-06-26 03:35:41 -07:00
Celes Hillyerd
bf9b7e65dd
revert: restore registry cache, investigate LE cert
2026-06-26 03:28:59 -07:00
Celes Hillyerd
b516d6805f
ci: fix remaining dockerhub-cache references in test/finalize
2026-06-26 03:27:25 -07:00
Celes Hillyerd
a0eaa7d6c8
ci: pull from Docker Hub directly, remove broken registry cache/mirrors
2026-06-26 03:27:12 -07:00
Celes Hillyerd
17a044de0f
ci: fix buildkit_config setting name (was buildkitd_config)
2026-06-26 03:21:46 -07:00
Celes Hillyerd
cf38e6aabc
ci: fix buildkitd_config YAML format for insecure registry
2026-06-26 03:16:00 -07:00
Celes Hillyerd
1ef016e652
ci: webhook fixed to internal URL
2026-06-26 03:11:05 -07:00
Celes Hillyerd
649e039164
ci: after repo repair
2026-06-26 03:10:48 -07:00
Celes Hillyerd
e5944dd755
ci: webhook recreated
2026-06-26 03:04:13 -07:00
Celes Hillyerd
3cc438abef
ci: webhook test
2026-06-26 03:01:50 -07:00
Celes Hillyerd
7d2cb93f50
ci: fix webhook after gitea url revert
2026-06-26 03:01:21 -07:00
Celes Hillyerd
00a3b2bd48
ci: test with minimal agent config
2026-06-26 03:00:11 -07:00
Celes Hillyerd
e1fe935ee4
ci: minimal agent config
2026-06-26 02:58:40 -07:00
Celes Hillyerd
0810699a9c
ci: gitea ROOT_URL uses ClusterIP
2026-06-26 02:55:39 -07:00
Celes Hillyerd
69a95c05d7
ci: dns debug
ci/woodpecker/push/test Pipeline failed
ci/woodpecker/push/build-3 unknown status
ci/woodpecker/push/build-1 unknown status
ci/woodpecker/push/build-2 unknown status
ci/woodpecker/push/finalize unknown status
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-06-26 02:54:37 -07:00
Celes Hillyerd
3d40a6ebff
ci: add host alias for gitea
ci/woodpecker/push/test Pipeline failed
ci/woodpecker/push/build-3 unknown status
ci/woodpecker/push/build-1 unknown status
ci/woodpecker/push/build-2 unknown status
ci/woodpecker/push/finalize unknown status
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-06-26 02:53:37 -07:00
Celes Hillyerd
6b470bab32
ci: clone via ClusterIP
ci/woodpecker/push/test Pipeline failed
ci/woodpecker/push/build-3 unknown status
ci/woodpecker/push/build-1 unknown status
ci/woodpecker/push/build-2 unknown status
ci/woodpecker/push/finalize unknown status
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-06-26 02:48:46 -07:00
Celes Renata
Celes Hillyerd