Root cause: DNS returns IPv6 (AAAA) records for registry.npmjs.org
first. Alpine musl tries IPv6, which hangs because the cluster has
no IPv6 routing. NODE_OPTIONS=--dns-result-order=ipv4first forces
Node/npm to use IPv4 addresses.
BuildKit sandbox has no network access for DNS resolution. Instead of
fighting it, run npm ci in a normal Woodpecker step (which has full
network), then COPY the whole frontend dir (including node_modules)
into the Docker build context. The Dockerfile just runs npm run build.
Root cause confirmed: BuildKit sandbox creates isolated network namespace
that cannot resolve DNS even though the pod itself can. The buildx_flags
--driver-opt network=host only affects the buildkit daemon creation, not
the actual build steps. Adding network: host passes --network=host to
docker buildx build, making RUN steps use host networking.
Root cause: CoreDNS pods were failing to forward external DNS queries
(registry.npmjs.org.celestium.life -> no next plugin). Restarted
CoreDNS deployment. Reverted Dockerfile to original clean state.
The BuildKit sandbox cannot resolve registry.npmjs.org DNS. Setting
networkMode=host in the worker config forces all RUN steps to use
the host network stack, which has confirmed internet access.
BuildKit sandbox DNS cannot resolve registry.npmjs.org. Using
--network=host on the RUN step lets npm use the host network stack
directly, bypassing the broken sandbox DNS.
BuildKit sandbox does not inherit cluster DNS, causing npm ci to fail
with EAI_AGAIN. Inject Google/Cloudflare DNS into resolv.conf before
running npm ci.
All pipeline files now specify clone.git.settings.remote with the
Gitea service ClusterIP (10.43.73.77:3000) instead of relying on
DNS resolution of gitea-service.git-server.svc.cluster.local.
This permanently fixes the intermittent DNS resolution failures
that occur when multiple build pipelines clone simultaneously.
Document Stages now shows time-filtered counts from /health endpoint
(consistent with Parsing/Extraction/Trends sections) with all-time
totals shown as a subtle subtitle from the SSE stream. Added "(last Xh)"
labels to all sections so the time window context is always visible.
_process_macro_classification never updated document status, leaving
macro_event docs stuck in parsed forever. Now marks them extracted on
success or extraction_failed on error.
- Replace setdefault() with explicit None checks in _normalize_extraction_data()
- Coerce null top-level fields (summary, novelty_score, confidence) to defaults
- Coerce null company fields (ticker, impact_score, impact_horizon, etc.) to defaults
- Filter out company entries with empty ticker after normalization
- Prevents schema validation failures when model returns null for required fields
- Strategy: Recreate (GPU can only be used by one pod at a time)
- nuextract-external targetPort: 8080 (nginx proxy port)
- served-model-name: numind/NuExtract3
- DB ai_agents updated to model=numind/NuExtract3 provider=vllm
- Ollama scaled to 0 — vLLM gets entire 16GB GPU
- NuExtract3 (9B) running at 80% GPU utilization
- Both OLLAMA_BASE_URL and VLLM_BASE_URL point to vLLM service
- Fixed gremlin-1 missing default route (no internet access)
- Removed GPU resource limit (time-slicing handles allocation)
- Removed init container (direct download works with internet fix)
- vLLM image mirrored to registry.celestium.life/stonks-oracle/vllm-openai
- Deployment uses Harbor image (Docker Hub IPv6 unreachable from cluster)
- All 3 pipelines use vllm-external.vllm-service.svc.cluster.local:2701
- K8s manifests at infra/kube-vllm/ and synced to ~/sources/kube/vllm
K8s vLLM deployment blocked by Docker Hub IPv6 connectivity issue.
Falling back to NixOS systemd vLLM service on esnixi.
K8s manifests remain in infra/kube-vllm/ for future use.
Sentiment Heatmap and Prediction Accuracy passed limit=500 but
the trends and recommendations endpoints enforce limit<=200,
causing a 422 validation error that made both panels show blank.
New ArgoCD stage 'stonks-live-math' running the dual-pipeline signal
engine against a separate Alpaca paper account for production validation.
- Namespace: stonks-live-math
- Database: stonks_live_math (Redis DB 3)
- Broker: PK64RS7NH24XPBI3IDEU3BB72Y (paper-api.alpaca.markets)
- Ingress: stonks-math-*.celestium.life
- Deploys alongside existing live stage (same image tag)
- Reports list page with type filter (daily/weekly)
- Report detail page with all sections: P&L, recommendation accuracy,
position performance table, risk metrics, model quality windows
- Executive summary card, validation warnings display
- Nav item under Trading group
- Routes: /reports and /reports/:id
- Add yAxisId='left' to vertical ReferenceLine (required by Recharts)
- Increase bottom margin from 40 to 70 for date label + legend space
- Legend uses verticalAlign='bottom' with 24px top padding
- Dockerfile now copies scripts/ directory into /app/scripts/
- Init container uses sh -c with fallback so missing script
doesn't crash the pod (graceful degradation until image rebuilds)
Inline Python with f-strings and colons broke Helm YAML parsing.
Moved to scripts/backfill_market_data.py and call it directly
from the init container command.
- Click 'Expand' button to view the trend chart fullscreen
- Press Escape or click outside to close
- Fullscreen uses full viewport height for better readability
- Green dashed vertical lines at 9:30 AM ET (market open)
- Red dashed vertical lines at 4:00 PM ET (market close)
- Markers only shown on intraday and 1d windows (hidden on 7d+)
- Chart content extracted into shared function to avoid duplication
Alpaca's current_price reflects extended hours trading which is more
current than Polygon's regular session close. Keep it as the display
price. Add polygon_price as a reference field in the API response.
Alpaca paper trading returns inaccurate current_price values.
The positions endpoint now uses the latest Polygon close from
market_snapshots and recomputes unrealized P&L from that.
1. Add backfill-market-data init container to scheduler deployment.
On vanilla start (< 50 bars in market_snapshots), fetches 90 days
of daily OHLCV from Polygon directly via asyncpg + httpx.
2. Fix trading engine dedup key placement. Previously the Redis dedup
key was set BEFORE evaluate_recommendation(), so recs skipped as
outside_trading_window were permanently deduped and never retried
when the market opened. Now the dedup key is only set AFTER the
decision for non-retryable outcomes (act, confidence_too_low, etc).
outside_trading_window skips are retryable.
After scale_up, restart the scheduler and wait for it to be ready
before other services start. The scheduler's init containers run
migrations, so this ensures tables exist before ingestion/extractor
pods try to query them.
Backend:
- GET /api/market/prices/{ticker} now returns { bars, range_90d }
with 90-day low/high computed from market_snapshots
- POST /api/market/backfill/{ticker} fetches 90 days of daily bars
from Polygon and inserts missing bars into market_snapshots
- POST /api/market/backfill-all does the same for all active tickers
Frontend:
- Right Y-axis domain scaled to 90-day min/max (with 3% padding)
- Green dashed reference line at 90-day high
- Red dashed reference line at 90-day low
- Labels show exact price on each reference line
- Default limit bumped to 200 bars
- Add initial capital input (toggle between broker balance or custom amount)
- Add reserve/active pool split slider (0-50%, default 20%)
- Backend accepts reserve_pct in reset request body
- Note in UI that Alpaca balance reset requires Alpaca dashboard
- Confirmation dialog shows exact capital and split being applied
Changed intraday_bars default from 1-hour bars to 15-minute bars.
This gives ~26 price points per trading day per ticker (6.5h market
× 4 bars/hour) instead of ~7 hourly bars. Limit raised to 100 to
accommodate the higher bar count.
The market_api sources were configured with endpoint='prev_bars' which
only fetches a single previous-day bar per ticker. Changed to
'intraday_bars' which fetches hourly bars for today from Polygon's
/v2/aggs/ticker/{ticker}/range/1/hour/{today}/{today} endpoint.
Updated: seed script, beta DB (50 sources), production DB (50 sources).
This gives ~7-8 hourly price bars per trading day per ticker instead
of 1 daily bar.
Only ~9 price bars per ticker (Polygon returns daily bars, not
intraday). Widened gap tolerance to 6h for intraday, 12h for 1d,
etc. Also skip time-range filtering when price data is sparse
(≤20 bars) to avoid showing no prices at all.
Was showing every data point timestamp. Now:
- Recharts generates max 8 evenly-spaced ticks
- Each tick shows 'Apr 29' in bold white + '2 PM' in gray
- All labels at -35° angle to avoid overlap
- Simplified tick component (no hour-boundary filtering needed)
Replaced string-based X-axis with numeric timestamp axis:
- Custom ChartXTick component renders hour marks at -35° angle
- New day boundaries shown in bold (e.g., 'Apr 29')
- Hour marks shown as '9:00 AM', '10:00 AM' etc.
- Tooltip shows full date+time on hover
- Direction timeline uses formatted timestamps
- Bottom margin increased to accommodate angled labels
Two fixes for missing intraday data:
1. Frontend: lifted selectedWindow state to page level so useTrendHistory
passes window param to the API. Previously fetched all windows with
limit=500 which exhausted the limit before reaching recent intraday
data. Now fetches only the selected window's data.
2. Scheduler: removed market-hours-only restriction from periodic
aggregation. Runs every 15 minutes 24/7 so intraday data is always
populated for backtesting regardless of market state.
The aggregation engine only ran when new documents were ingested,
leaving intraday trend data stale for long periods. Now the scheduler
enqueues all 50 tickers for re-aggregation every ~15 minutes during
US market hours (Mon-Fri, 6:30 AM - 1:30 PM PT). This ensures
continuous intraday trend updates based on existing signals and
market price changes.
Intraday was showing only 12h of data (9 sparse points). Widened to
24h to show a full day of intraday trend history. Also widened 1d
from 24h to 48h for better context.
Celes Renata