Celes Renata
987b32d2ff
fix: reduce correlation window 30d→7d to prevent OOM (132k→~30k rows)
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 21:46:54 +00:00
Celes Renata
e1d1c70d59
ci: trigger pipeline after Woodpecker OAuth fix
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 15:36:17 +00:00
Celes Renata
4f021511fa
fix: move correlation matrix to background task — prevents startup timeout/OOM
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 15:23:22 +00:00
Celes Renata
6bde9d2726
fix: increase trading-engine memory limit 512Mi→1024Mi (OOMKilled by correlation matrix)
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-21 15:20:38 +00:00
Celes Renata
620a8f0d91
fix: add specialist health check, update default namespace to stonks-oracle
Build and Push / lint-and-test (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Canceled after 0s
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Canceled after 0s
Build and Push / build-dashboard (push) Canceled after 0s
Build and Push / build-superset (push) Canceled after 0s
Build and Push / integration-test (push) Canceled after 0s
Build and Push / beta-gate (push) Canceled after 0s
2026-07-13 09:17:23 +00:00
Celes Renata
f9284ce0a8
docs: update steering files with CI DNS fix, specialist build, and new pitfalls
2026-07-13 09:12:50 +00:00
Celes Renata
d4f1b529a5
fix: add missing specialist image build step to CI pipeline
2026-07-13 08:08:23 +00:00
Celes Renata
9617b088bd
fix: allow node_modules in Docker context (pre-staged by CI step)
2026-07-13 07:40:19 +00:00
Celes Renata
c0d7f2f438
fix: pin registry.npmjs.org IPv4 in /etc/hosts to bypass broken DNS
2026-07-13 07:31:38 +00:00
Celes Renata
7229273461
fix: reduce npm maxsockets to 3, add retries for EAI_AGAIN DNS failures
2026-07-13 07:27:51 +00:00
Celes Renata
d18a63aab5
fix: remove invalid npm config, keep http logging for diagnostics
2026-07-13 07:23:40 +00:00
Celes Renata
7df45f7e27
fix: use map format for Woodpecker environment vars
2026-07-13 07:19:47 +00:00
Celes Renata
0d4a812e3d
fix: add DNS diagnostics to npm-install-dashboard CI step
2026-07-13 06:58:43 +00:00
Celes Renata
2fff553518
fix: force IPv4 for npm — cluster has no IPv6 egress
...
Root cause: DNS returns IPv6 (AAAA) records for registry.npmjs.org
first. Alpine musl tries IPv6, which hangs because the cluster has
no IPv6 routing. NODE_OPTIONS=--dns-result-order=ipv4first forces
Node/npm to use IPv4 addresses.
2026-07-13 06:41:46 +00:00
Celes Renata
2d516a725d
fix: pre-stage npm ci outside BuildKit sandbox
...
BuildKit sandbox has no network access for DNS resolution. Instead of
fighting it, run npm ci in a normal Woodpecker step (which has full
network), then COPY the whole frontend dir (including node_modules)
into the Docker build context. The Dockerfile just runs npm run build.
2026-07-13 05:55:41 +00:00
Celes Renata
cdad58ae28
fix: add network=host to dashboard buildx build step
...
Root cause confirmed: BuildKit sandbox creates isolated network namespace
that cannot resolve DNS even though the pod itself can. The buildx_flags
--driver-opt network=host only affects the buildkit daemon creation, not
the actual build steps. Adding network: host passes --network=host to
docker buildx build, making RUN steps use host networking.
2026-07-13 05:52:00 +00:00
Celes Renata
9645d9c261
fix: revert Dockerfile hacks, restart CoreDNS to fix DNS resolution
...
Root cause: CoreDNS pods were failing to forward external DNS queries
(registry.npmjs.org.celestium.life -> no next plugin). Restarted
CoreDNS deployment. Reverted Dockerfile to original clean state.
2026-07-13 05:20:48 +00:00
Celes Renata
05e6f21a16
fix: set BuildKit worker networkMode=host for dashboard build
...
The BuildKit sandbox cannot resolve registry.npmjs.org DNS. Setting
networkMode=host in the worker config forces all RUN steps to use
the host network stack, which has confirmed internet access.
2026-07-13 05:02:35 +00:00
Celes Renata
d762a2b76f
fix: use RUN --network=host for npm ci in BuildKit
...
BuildKit sandbox DNS cannot resolve registry.npmjs.org. Using
--network=host on the RUN step lets npm use the host network stack
directly, bypassing the broken sandbox DNS.
2026-07-13 04:47:37 +00:00
Celes Renata
46d7699ed0
fix: add explicit DNS resolvers to frontend Dockerfile for BuildKit
...
BuildKit sandbox does not inherit cluster DNS, causing npm ci to fail
with EAI_AGAIN. Inject Google/Cloudflare DNS into resolv.conf before
running npm ci.
2026-07-13 04:38:59 +00:00
Celes Renata
13729bda98
ci: retry (DNS resolution failure in BuildKit)
2026-07-13 04:19:28 +00:00
Celes Renata
b5deeaa0d5
ci: retry build (npm ci hung on previous attempt)
2026-07-13 03:07:01 +00:00
Celes Renata
0aa72cbce1
fix: add numpy and scikit-learn to requirements for v3 calibrators
2026-07-13 02:19:44 +00:00
Celes Renata
a72f336ad1
feat: Intelligence Pipeline v3 — full implementation
...
Multi-stage evidence-grounded inference architecture replacing the
monolithic 9B model extraction pipeline. CPU-first specialist services
handle routine extraction while the 9B vLLM model is preserved for
semantic adjudication of ambiguous cases.
Key components:
- Capability-aware inference gateway (OpenAI-compatible + Ollama)
- Endpoint registry with DB migrations and REST API
- Sentence-aware document segmenter (property tests)
- Deterministic financial parsing with offset integrity
- Symbol resolution with ambiguity detection
- Specialist service (GLiNER2, dynamic batching, K8s deployment)
- Company-specific sentiment (FinBERT, calibration)
- Retrieval-based novelty and duplicate detection
- Confidence calibration pipeline
- Deterministic routing engine (property tests)
- 9B adjudication layer with VRAM gating
- Stock-specific impact model (features, labels, baseline, trained)
- Pipeline orchestrator (state machine, queues, leases, feature flags)
- Bounded parallelism (async workers, semaphore, load shedding)
- Observability (tracing, metrics, alerts)
- Compatibility adapter (v3→v2 golden mapping tests)
- Shadow/canary promotion framework
- Active learning and fine-tuning pipeline
Test results: 1,161 tests pass, ruff lint clean.
All 282 spec tasks completed.
2026-07-13 02:14:59 +00:00
Celes Renata
84634a365e
feat: deep pipeline health check script
...
Checks 12 aspects of the running deployment:
- Pod health (replicas, restarts, OOMKills)
- PostgreSQL connectivity & stats
- Redis connectivity & queue backlog
- MinIO connectivity
- vLLM model availability
- Alpaca broker account status
- Pipeline flow (ingestion→extraction→aggregation→recommendation)
- Model quality gate freshness
- Paper trading portfolio status
- API endpoint health
- Resource usage
- End-to-end latency (doc processing, trend/rec freshness)
Usage: ./scripts/pipeline_health_check.sh [namespace]
2026-07-11 22:35:04 +00:00
Celes Renata
2f966b1caf
fix: scheduler memory 128Mi→256Mi, fix backfill script SQL param bug
2026-07-11 17:23:52 +00:00
Celes Renata
81ed2f0803
fix(ci): bypass DNS for clone — use Gitea ClusterIP directly
...
All pipeline files now specify clone.git.settings.remote with the
Gitea service ClusterIP (10.43.73.77:3000) instead of relying on
DNS resolution of gitea-service.git-server.svc.cluster.local.
This permanently fixes the intermittent DNS resolution failures
that occur when multiple build pipelines clone simultaneously.
2026-07-11 05:36:41 +00:00
Celes Renata
5bd5c521dc
ci: dns debug
2026-07-11 05:28:31 +00:00
Celes Renata
9cbb559ac7
ci: retry - coredns permanent fix (dnsPolicy:None + forward 192.168.42.1)
2026-07-11 04:39:28 +00:00
Celes Renata
b13232a21f
ci: retry after reverting coredns and woodpecker dns to original config
2026-07-11 04:36:20 +00:00
Celes Renata
0cd052a0a7
ci: retry - restored search domains, ndots:2, timeout:10, attempts:5
2026-07-11 04:33:41 +00:00
Celes Renata
a2f4398f8d
ci: retry - dns config with ndots:3, timeout:5, attempts:5
2026-07-11 04:32:09 +00:00
Celes Renata
76e06d5951
ci: retry after scaling coredns to 4 replicas across all nodes
2026-07-10 23:25:16 +00:00
Celes Renata
96eabf595b
ci: retry after adding 192.168.42.1 to woodpecker DNS config
2026-07-10 23:21:18 +00:00
Celes Renata
430554fe0c
ci: retry after coredns fix + woodpecker agent restart
2026-07-10 23:18:59 +00:00
Celes Renata
853b85a2af
ci: retry after coredns forward fix
2026-07-10 23:14:18 +00:00
Celes Renata
1d104187f4
ci: retry after transient DNS failure
2026-07-10 23:11:32 +00:00
Celes Renata
d018c75182
fix: OpsPipeline dashboard — use time-filtered doc stages, add time labels
...
Document Stages now shows time-filtered counts from /health endpoint
(consistent with Parsing/Extraction/Trends sections) with all-time
totals shown as a subtle subtitle from the SSE stream. Added "(last Xh)"
labels to all sections so the time window context is always visible.
2026-07-10 22:14:56 +00:00
Celes Renata
ed474c521a
fix: update document status after macro classification
...
_process_macro_classification never updated document status, leaving
macro_event docs stuck in parsed forever. Now marks them extracted on
success or extraction_failed on error.
2026-07-10 22:09:02 +00:00
Celes Renata
e8e348d478
fix: update test — empty ticker now filtered out instead of failing extraction
2026-07-10 20:43:46 +00:00
Celes Renata
d9fbd5b660
fix: handle null fields from NuExtract3 in extraction normalization
...
- Replace setdefault() with explicit None checks in _normalize_extraction_data()
- Coerce null top-level fields (summary, novelty_score, confidence) to defaults
- Coerce null company fields (ticker, impact_score, impact_horizon, etc.) to defaults
- Filter out company entries with empty ticker after normalization
- Prevents schema validation failures when model returns null for required fields
2026-07-10 20:38:50 +00:00
Celes Renata
ca712ad4a0
fix: pipeline health — stuck docs, price fallback, sentiment normalization, signal-engine scale, quality gate
...
- Scheduler: lower stale threshold 240→30 min, batch limit 100→500, TTL 14400→3600
- Prediction snapshot: add 24h market_snapshots time-window fallback
- Aggregation: add normalize_impact_scores() z-score normalization
- Helm: signal-engine replicas → 0 (idle when dual pipeline disabled)
- Quality gate: max_snapshot_age_hours 24→48
- Add backfill script for NULL price_at_prediction snapshots
- Add PBT bug condition and preservation tests (14 tests)
2026-07-10 20:16:01 +00:00
Celes Renata
a4f51c00e1
fix: beta broker keys in values-beta.yaml (ArgoCD source of truth)
2026-07-09 14:04:59 +00:00
Celes Renata
cf93e2d547
chore: trigger CI rebuild for dashboard fix
2026-07-06 04:06:06 +00:00
Celes Renata
88a6c8f9ee
fix: beta broker keys — use prod Alpaca paper account
2026-07-06 02:46:29 +00:00
Celes Renata
af7b654fb7
fix: vLLM deployment — Recreate strategy, correct service port, single model name
...
- Strategy: Recreate (GPU can only be used by one pod at a time)
- nuextract-external targetPort: 8080 (nginx proxy port)
- served-model-name: numind/NuExtract3
- DB ai_agents updated to model=numind/NuExtract3 provider=vllm
2026-07-04 05:34:04 +00:00
Celes Renata
a322e00659
feat: vLLM serving numind/NuExtract3 on full GPU, ollama disabled
...
- Ollama scaled to 0 — vLLM gets entire 16GB GPU
- NuExtract3 (9B) running at 80% GPU utilization
- Both OLLAMA_BASE_URL and VLLM_BASE_URL point to vLLM service
- Fixed gremlin-1 missing default route (no internet access)
- Removed GPU resource limit (time-slicing handles allocation)
- Removed init container (direct download works with internet fix)
2026-07-03 17:57:19 +00:00
Celes Renata
ecade0dd52
feat: vLLM in K8s via Harbor mirror, pipelines point to internal svc
...
- vLLM image mirrored to registry.celestium.life/stonks-oracle/vllm-openai
- Deployment uses Harbor image (Docker Hub IPv6 unreachable from cluster)
- All 3 pipelines use vllm-external.vllm-service.svc.cluster.local:2701
- K8s manifests at infra/kube-vllm/ and synced to ~/sources/kube/vllm
2026-07-03 17:02:31 +00:00
Celes Renata
3a9894cd03
fix: revert VLLM to bare-metal 192.168.42.254:2701
...
K8s vLLM deployment blocked by Docker Hub IPv6 connectivity issue.
Falling back to NixOS systemd vLLM service on esnixi.
K8s manifests remain in infra/kube-vllm/ for future use.
2026-07-03 11:40:36 +00:00
Celes Renata
4ac10ab4f6
fix: point VLLM_BASE_URL to K8s internal vllm-service
...
vLLM now runs in vllm-service namespace with metrics proxy,
accessible at vllm-external.vllm-service.svc.cluster.local:2701
2026-07-03 11:38:24 +00:00
Celes Renata
3996ec6ef5
feat: switch models — ollama qwen3.5:4b-fast, vllm numind/NuExtract3
...
- Ollama: qwen3.5:4b-fast with NUM_PARALLEL=8, num_ctx=16384 (~6GB VRAM)
- vLLM: numind/NuExtract3 at 192.168.42.254:2701, gpu-memory-util=0.45
- All 3 pipelines (beta/paper/live) updated with correct endpoints
- Ollama proxy at 10.1.1.12:2701 (K8s ollama-metrics)
- vLLM proxy at 192.168.42.254:2701 (NixOS nginx)
2026-07-03 11:15:28 +00:00
Celes Renata
d1533ed31d
chore: test CI after CoreDNS restart
2026-07-03 10:17:38 +00:00
Celes Renata
02eb8e619f
chore: trigger CI rebuild
2026-07-03 09:55:37 +00:00
Celes Renata
a6aceebe18
fix: dashboards limit exceeds API max (500→200)
...
Sentiment Heatmap and Prediction Accuracy passed limit=500 but
the trends and recommendations endpoints enforce limit<=200,
causing a 422 validation error that made both panels show blank.
2026-07-03 09:35:55 +00:00
Celes Renata
14a9b4fcc1
fix: ops pipeline — rate limit, validation cycle, snapshots, config query, rejection reason
...
- Polygon rate limit env-configurable, default 5 (free tier)
- v3_engine_enabled reads from JSONB config column correctly
- Validation cycle (outcome eval + metrics) wired into scheduler hourly
- Daily portfolio/risk snapshots after 16:30 ET with idempotency
- Prediction snapshot price fallback from positions table
- Order rejection_reason + rejected_at persisted on insert
- Lake-publisher scaled to 0 replicas (redundant)
- Test fix: isolate per-type rate limit tests from Polygon global
2026-07-03 08:29:24 +00:00
Celes Renata
b70304ad6c
fix: correct Alpaca paper trading credentials for beta
2026-06-30 15:24:30 +00:00
Celes Renata
fdb54b77ce
fix: update Redis password, Ollama URL, and broker keys for new cluster
2026-06-28 22:03:26 +00:00
Celes Renata
106b07c0f0
fix: ruff lint fixes for v3 test files
2026-06-27 12:32:52 +00:00
Celes Renata
b4bf0f2361
feat: math core v3 engine upgrade
2026-06-27 12:21:41 +00:00
Celes Renata
5ed9f001b1
docs: add bash exclamation mark pitfall to steering file
2026-05-02 08:33:40 +00:00
Celes Renata
18f857e1a3
feat: add live-math stage with dedicated paper trading account
...
New ArgoCD stage 'stonks-live-math' running the dual-pipeline signal
engine against a separate Alpaca paper account for production validation.
- Namespace: stonks-live-math
- Database: stonks_live_math (Redis DB 3)
- Broker: PK64RS7NH24XPBI3IDEU3BB72Y (paper-api.alpaca.markets)
- Ingress: stonks-math-*.celestium.life
- Deploys alongside existing live stage (same image tag)
2026-05-02 08:19:47 +00:00
Celes Renata
61bcb5aa57
ci: add signal-engine image build to Woodpecker pipeline
2026-05-02 07:43:59 +00:00
Celes Renata
f468e30af0
feat: implement dual-pipeline signal engine service
...
New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.
Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading
Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant
Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)
2026-05-02 07:32:26 +00:00
Celes Renata
7e2343ec2c
feat: add Reports page and detail view to frontend
...
- Reports list page with type filter (daily/weekly)
- Report detail page with all sections: P&L, recommendation accuracy,
position performance table, risk metrics, model quality windows
- Executive summary card, validation warnings display
- Nav item under Trading group
- Routes: /reports and /reports/:id
2026-05-02 00:27:34 +00:00
Celes Renata
3a8c6f6c80
fix: add CACHE_BUST ARG to Dockerfile + build args to force fresh COPY layers per commit
2026-05-01 23:23:49 +00:00
Celes Renata
139fbd6342
fix: add no_cache to all Woodpecker service builds to prevent stale COPY layers
2026-05-01 23:11:21 +00:00
Celes Renata
63b62c5e9f
chore: RUN echo cache-bust to force buildx layer invalidation
2026-05-01 22:57:29 +00:00
Celes Renata
3acfdfd11f
chore: move cache-bust before COPY to invalidate buildx layer cache
2026-05-01 22:48:12 +00:00
Celes Renata
ab212db8be
chore: bust Docker buildx cache for services/ layer
2026-05-01 22:39:38 +00:00
Celes Renata
7102358f51
chore: retrigger CI (bust Docker buildx cache for services/ layer)
2026-05-01 22:32:23 +00:00
Celes Renata
bc077bfcc8
feat: trading feedback engine — periodic performance reports with AI summarization
...
- Migration 038: trading_reports table + report-summarizer agent seed
- 6 reporting modules: models, collector, sections, validator, summarizer, generator
- API endpoints: GET /api/reports (paginated, filterable), GET /api/reports/{id}
- Frontend hooks: useReports, useReport with TanStack Query
- Scheduler: daily (after 16:30 ET) and weekly (Saturday) report triggers
- Redis queue consumer for async report generation with retry/dedup
- 5 property-based tests (chunking, serialization, validation, accuracy, deltas)
- 109 unit/integration tests across all modules
- 6 frontend hook tests with MSW mocks
2026-05-01 22:13:09 +00:00
Celes Renata
376fcb4bb4
feat: add 12 integrity check saved queries for SQL Explorer (migration 037)
...
Prefixed with ⚕ for easy identification. Checks: duplicate snapshots,
orphaned links, evidence count mismatches, contribution score sums,
canonical key consistency, out-of-range values, unmatched snapshots,
zero evidence rate, duplicate count mismatches, missing prices,
outcome integrity, and pipeline health summary.
2026-05-01 19:20:56 +00:00
Celes Renata
cc21fd9e8f
feat: add 15 validation saved queries for SQL Explorer (migration 036)
...
Prediction overview, confidence distribution, evidence dedup quality,
source/catalyst breakdown, win rate by ticker/horizon, model quality
timeline, quality gate status, high-duplicate predictions, excess vs SPY.
2026-05-01 19:15:54 +00:00
Celes Renata
affb65d7f4
fix: point VLLM_BASE_URL to Ollama (10.1.1.12:2701) and update model name
...
vLLM service on 192.168.42.254:8000 was down causing extraction failures.
Also updated ai_agents table to use ollama provider directly.
2026-05-01 18:56:55 +00:00
Celes Renata
32535540fe
feat: add no-premature-simplification rule to steering docs
2026-05-01 04:32:05 +00:00
Celes Renata
751cce0509
feat: add documentation maintenance rules to steering, update migration number to 036
2026-05-01 04:29:52 +00:00
Celes Renata
0732894414
fix: widen horizon/mode columns in prediction_snapshots to VARCHAR(50)
...
time_horizon values like 'position_30d_90d (proj:30d)' exceed VARCHAR(20)
2026-05-01 04:27:19 +00:00
Celes Renata
d9110d03a6
fix: use di.document_id instead of dir.document_id in evidence query
...
document_impact_records has intelligence_id, not document_id.
The document_id lives on document_intelligence.
2026-05-01 04:06:55 +00:00
Celes Renata
76f6bd5677
fix: quote reserved keyword 'window' in prediction_snapshots SQL
2026-05-01 03:40:48 +00:00
Celes Renata
32d290bea7
fix: ruff lint — sort imports, remove unused GateThresholdResult import
2026-05-01 03:08:21 +00:00
Celes Renata
7fcc8a6c07
feat: model validation, calibration, and signal quality layer
...
- Migration 035: prediction_snapshots, prediction_outcomes, signal_evidence_links, model_metric_snapshots tables + SQL views
- Prediction snapshot writer with canonical evidence keys, duplicate detection, contribution scores
- Outcome evaluator across 5 horizons (1h, 6h, 1d, 7d, 30d)
- Metrics engine: ECE, Brier score, IC, Rank IC, benchmark comparison
- Attribution engine: per-source, per-catalyst, per-layer performance
- Calibration engine: Bayesian shrinkage source reliability
- Quality gate for live trading eligibility with configurable thresholds
- 7 new /api/validation/* endpoints
- Upgraded OpsModel dashboard with validation tab
- Enhanced recommendation display with calibration context
- Backtest replay validation mode
- 86 Python tests (unit + property-based), 179 frontend tests passing
2026-05-01 03:04:58 +00:00
Celes Renata
5d2ffd9163
fix: vertical market markers need yAxisId + move legend below date labels
...
- Add yAxisId='left' to vertical ReferenceLine (required by Recharts)
- Increase bottom margin from 40 to 70 for date label + legend space
- Legend uses verticalAlign='bottom' with 24px top padding
2026-04-30 23:31:55 +00:00
Celes Renata
6169efdc89
fix: copy scripts/ into Docker image + graceful backfill fallback
...
- Dockerfile now copies scripts/ directory into /app/scripts/
- Init container uses sh -c with fallback so missing script
doesn't crash the pod (graceful degradation until image rebuilds)
2026-04-30 23:18:43 +00:00
Celes Renata
c42f2223d8
fix: move backfill logic to standalone script (fixes YAML parse error)
...
Inline Python with f-strings and colons broke Helm YAML parsing.
Moved to scripts/backfill_market_data.py and call it directly
from the init container command.
2026-04-30 23:01:20 +00:00
Celes Renata
1f08820f11
feat: fullscreen chart expand + market open/close vertical markers
...
- Click 'Expand' button to view the trend chart fullscreen
- Press Escape or click outside to close
- Fullscreen uses full viewport height for better readability
- Green dashed vertical lines at 9:30 AM ET (market open)
- Red dashed vertical lines at 4:00 PM ET (market close)
- Markers only shown on intraday and 1d windows (hidden on 7d+)
- Chart content extracted into shared function to avoid duplication
2026-04-30 22:44:00 +00:00
Celes Renata
2f2ea65fb4
fix: keep Alpaca price as primary (includes after-hours), add polygon_price field
...
Alpaca's current_price reflects extended hours trading which is more
current than Polygon's regular session close. Keep it as the display
price. Add polygon_price as a reference field in the API response.
2026-04-30 22:35:22 +00:00
Celes Renata
9a60ce127b
fix: overlay Polygon prices on positions instead of stale Alpaca prices
...
Alpaca paper trading returns inaccurate current_price values.
The positions endpoint now uses the latest Polygon close from
market_snapshots and recomputes unrealized P&L from that.
2026-04-30 22:32:21 +00:00
Celes Renata
414f476620
fix: auto-backfill market data on startup + fix trading dedup bug
...
1. Add backfill-market-data init container to scheduler deployment.
On vanilla start (< 50 bars in market_snapshots), fetches 90 days
of daily OHLCV from Polygon directly via asyncpg + httpx.
2. Fix trading engine dedup key placement. Previously the Redis dedup
key was set BEFORE evaluate_recommendation(), so recs skipped as
outside_trading_window were permanently deduped and never retried
when the market opened. Now the dedup key is only set AFTER the
decision for non-retryable outcomes (act, confidence_too_low, etc).
outside_trading_window skips are retryable.
2026-04-30 14:08:54 +00:00
Celes Renata
a5f2bcde55
fix: set MARKET_DATA_BASE_URL to polygon in base values.yaml
2026-04-30 07:20:13 +00:00
Celes Renata
34ffdad00c
fix: wait for scheduler migrations after factory reset scale-up
...
After scale_up, restart the scheduler and wait for it to be ready
before other services start. The scheduler's init containers run
migrations, so this ensures tables exist before ingestion/extractor
pods try to query them.
2026-04-30 07:01:01 +00:00
Celes Renata
601b85764b
fix: mount stonks-market-secrets on query-api for backfill endpoint
2026-04-30 06:33:26 +00:00
Celes Renata
51b6f3d34a
ci: trigger rebuild
2026-04-30 06:24:05 +00:00
Celes Renata
bddaf44ffc
feat: 90-day price range Y-axis scaling with breakthrough annotations
...
Backend:
- GET /api/market/prices/{ticker} now returns { bars, range_90d }
with 90-day low/high computed from market_snapshots
- POST /api/market/backfill/{ticker} fetches 90 days of daily bars
from Polygon and inserts missing bars into market_snapshots
- POST /api/market/backfill-all does the same for all active tickers
Frontend:
- Right Y-axis domain scaled to 90-day min/max (with 3% padding)
- Green dashed reference line at 90-day high
- Red dashed reference line at 90-day low
- Labels show exact price on each reference line
- Default limit bumped to 200 bars
2026-04-30 06:09:05 +00:00
Celes Renata
5209cc522e
feat: enhanced paper trading reset with capital and reserve controls
...
- Add initial capital input (toggle between broker balance or custom amount)
- Add reserve/active pool split slider (0-50%, default 20%)
- Backend accepts reserve_pct in reset request body
- Note in UI that Alpaca balance reset requires Alpaca dashboard
- Confirmation dialog shows exact capital and split being applied
2026-04-30 05:49:38 +00:00
Celes Renata
fa18b1a7c2
fix: populate broker and market API secrets in base values.yaml
...
ArgoCD was syncing empty secrets back over manual patches.
Credentials now match stonks-paper (same Alpaca paper + Polygon keys).
2026-04-30 05:05:43 +00:00
Celes Renata
4b254adad2
fix: scale ingestion and extractor to 1 replica in prod
2026-04-30 05:04:20 +00:00
Celes Renata
af2e554edd
fix: switch ai_agents seed from vllm to ollama qwen3.5:9b-fast
...
Live DB already updated via direct SQL. This ensures future
DB rebuilds and new deployments use ollama as the default provider.
2026-04-30 04:46:04 +00:00
Celes Renata
facce5dbb5
fix: add TTL and backoffLimit to market-hours-check jobs
...
Prevents accumulation of completed/failed verification jobs.
- ttlSecondsAfterFinished: 3600 (auto-cleanup after 1h)
- backoffLimit: 0 (no retries on intentional market-hours failure)
2026-04-30 04:33:50 +00:00
Celes Renata
2f2c0d24f6
fix: remove unused columns constant (TS6133)
2026-04-30 03:21:32 +00:00
Celes Renata
861423c1e3
feat: make ticker clickable on positions page — links to company detail
...
Ticker column now links to /companies/{id} using a ticker→company ID
lookup. Falls back to plain text if company not found.
2026-04-30 03:15:22 +00:00