Commit Graph

302 Commits

Author SHA1 Message Date
Celes Renata 18eb150c75 fix: confidence formula now uses unique doc count + signal agreement instead of raw signal count — prevents 99.9% inflation 2026-04-16 23:48:05 +00:00
Celes Renata 1a5fb2e36a fix: recovery sweep touches updated_at after re-enqueue to prevent duplicate flooding every 5 min 2026-04-16 19:13:10 +00:00
Celes Renata 693d9e0d60 fix: reduce LLM timeouts — truncate docs to 8k/6k chars, cut num_predict 16k→4k, tighten prompts, trim anti-hallucination rules 2026-04-16 18:56:11 +00:00
Celes Renata 3a856cf6ff fix: reduce Ollama timeout from 300s to 240s (4 min) 2026-04-16 18:43:50 +00:00
Celes Renata 60cfb7618e fix: recovery sweep skips docs that already have global_events — prevents re-enqueue loop 2026-04-16 18:27:21 +00:00
Celes Renata 1043710b6d fix: track last_published_at per source to avoid re-fetching same articles — applies to both news_api and macro_news 2026-04-16 18:12:12 +00:00
Celes Renata 513310abba fix: stop tagging all macro_news articles as macro_event — default to article, let extractor reclassify. Also reduced fetch limit to 20 and cadence to 30min 2026-04-16 18:09:50 +00:00
Celes Renata f83577480f fix: alternate extractor between macro and extraction queues (1:2 ratio) to prevent starvation 2026-04-16 17:45:25 +00:00
Celes Renata 2440cddd37 feat: add comprehensive table retention cleanup to scheduler — 10 tables with per-table retention windows 2026-04-16 15:56:50 +00:00
Celes Renata 63287903d0 feat: wire up stop levels, circuit breaker daily loss, profit-taking, real portfolio/decisions/history endpoints 2026-04-16 15:52:46 +00:00
Celes Renata 1329df0bbf feat: sell execution, correlation matrix from market data, US market holiday awareness
- Sell path: looks up existing position, sells full quantity, returns proceeds to pool
- Correlation matrix: computed from 30-day market_snapshots on startup + every 5min
- Holidays: 10 major US market holidays for 2026 checked in trading window functions
2026-04-16 15:36:49 +00:00
Celes Renata 2e77cf32fd fix: critical — track capital properly: load invested positions on startup, deduct on act, sync every 5min 2026-04-16 15:29:28 +00:00
Celes Renata 9a8d36068a fix: convert Decimal to float in API responses instead of string — fixes positions page crash 2026-04-16 15:25:40 +00:00
Celes Renata 354c3d484a fix: fetch current prices from market_snapshots before evaluating recommendations — fixes 'Invalid current price' skip 2026-04-16 15:17:49 +00:00
Celes Renata 2a6aac47a6 fix: add decision logging to trading engine, flushed 103k stale dedup keys 2026-04-16 15:12:58 +00:00
Celes Renata c114e77b1c fix: limit recommendation poll to 50 per cycle to prevent 85k-rec processing stall, add poll logging 2026-04-16 15:05:26 +00:00
Celes Renata 136b149000 fix: add logging config to trading engine, add /api/trading/debug diagnostic endpoint 2026-04-16 14:55:42 +00:00
Celes Renata 6bab199159 fix: trend_windows now upserts instead of accumulating (7.5GB→4MB), add competitive signal retention cleanup 2026-04-16 14:32:24 +00:00
Celes Renata 58a8726306 feat: add paper trading capital controls — API endpoint + UI with presets, fix status/metrics to read real state, fix migration duplicates 2026-04-16 14:06:30 +00:00
Celes Renata 14e411daf9 fix: trading status and metrics endpoints now read real portfolio state instead of hardcoded zeros 2026-04-16 14:02:38 +00:00
Celes Renata 5cc64498c0 fix: skip already-propagated docs in aggregation, limit to last hour + 10 docs instead of 50 2026-04-16 09:35:11 +00:00
Celes Renata 540d54c3f7 feat: scale aggregation to 4 replicas across cluster nodes 2026-04-16 09:26:22 +00:00
Celes Renata f0887afd9b fix: cap competitive signals to 500 most recent per window to prevent 67k row aggregation bottleneck 2026-04-16 09:24:31 +00:00
Celes Renata c5e9644f6f fix: guard EventSource for jsdom test env, add rate-limits MSW handler 2026-04-16 08:19:53 +00:00
Celes Renata 58a05ca322 feat: add SSE stream for live pipeline status, add all 10 queues + DLQs, configure nginx for SSE 2026-04-16 08:15:44 +00:00
Celes Renata 0b1640abb9 feat: add status colors for document pipeline stages, add status filter to documents page 2026-04-16 08:06:12 +00:00
Celes Renata 87579d68da fix: add stale document recovery sweep to scheduler, re-enqueues orphaned parsed docs every 5 min 2026-04-16 07:59:30 +00:00
Celes Renata cdc825619e feat: add live queue depths to pipeline health API and dashboard 2026-04-16 07:49:07 +00:00
Celes Renata 8050f4a03b chore: mark all Phase 2 tasks (27-37) as complete
All 152 tasks across both phases are now marked complete:
- Phase 1 (1-26): pure computation modules, property tests, API, frontend, infra
- Phase 2 (27-37): live decision loop, stop-loss monitor, performance metrics,
  risk tier scheduler, rebalancer, notification dispatch, backtest replay,
  real DB connections, paper trading config, integration tests
2026-04-16 07:27:35 +00:00
Celes Renata 0ee7f26633 feat: raise market_api rate to 20/min, add global Polygon cap at 45/min, add rate-limit API + watchlist warning 2026-04-16 07:26:10 +00:00
Celes Renata 0b3ab4ed90 feat: add 11 new saved queries, fix window quoting and cross-table join in samples 2026-04-16 07:14:44 +00:00
Celes Renata f1e32e9186 fix: add round(double precision, integer) overload so ad-hoc queries work without ::numeric casts 2026-04-16 07:10:23 +00:00
Celes Renata 981e16a27f fix: quote reserved keyword 'window' in trend query, add Top Movers sample, make seed upsert 2026-04-16 06:25:38 +00:00
Celes Renata 1ccea17600 fix: tooltip type inference for Recharts strict mode 2026-04-16 06:01:29 +00:00
Celes Renata 79a85723b6 feat: rich tooltips in SQL Explorer charts show all row values
Hover over any bar, line point, or scatter dot to see every column
value for that data point. The Y-axis column is highlighted in
brand color, X-axis in white, and other columns in gray. Works
for all chart types (bar, line, scatter, auto).
2026-04-16 05:57:06 +00:00
Celes Renata b43ad88f5d feat: auto-chart detection in SQL Explorer
Adds an ' Auto' button that analyzes query results and picks the
best chart type and column mapping:

- Date/time column + numeric → line chart (time series)
- Categorical + numeric → bar chart (categories)
- Two numeric columns → scatter plot
- Shows detected type and column names as a label

Click Auto, run any query, and it figures out the rest.
2026-04-16 05:52:41 +00:00
Celes Renata 7fefc65692 chore: remove runmefirst.sh from repo, add to gitignore
Deploy scripts live on gremlin-1 at ~/sources/kube/stonks-oracle/,
not in the git repo. They reference local secret files and should
not be version controlled.
2026-04-16 05:39:56 +00:00
Celes Renata 328cb0de28 fix: SQL Explorer chart parses string values as floats
The pg-query API returns all values as strings. The chart builder
was using Number() which returns NaN for non-numeric strings.
Now uses parseFloat with NaN fallback to 0.
2026-04-16 05:33:53 +00:00
Celes Renata 1107d34027 fix: SQL Explorer handles comments and shows descriptive errors
- Strip SQL comments (-- and /* */) before checking for SELECT,
  so queries with leading comments don't get rejected
- Show the actual error detail from the API response instead of
  generic 'API error 400' in the SQL Explorer UI
2026-04-16 05:25:45 +00:00
Celes Renata d28787a8ee fix: add unique constraint on saved_queries.name to prevent duplicates
The migration ran on every deploy, inserting duplicate queries each
time (96 instead of 12). Added UNIQUE constraint on name and changed
ON CONFLICT to reference it. Cleaned up 84 duplicates in DB.
2026-04-16 05:16:18 +00:00
Celes Renata a3b2e97c2c fix: allow SMTP port 587 egress for Gmail notifications
The trading engine network policy only allowed egress on ports 443
(HTTPS) and 53 (DNS). Gmail SMTP uses port 587 (STARTTLS), causing
'Network is unreachable' when sending notifications.
2026-04-16 05:04:08 +00:00
Celes Renata c4666c071b feat: wire Gmail SMTP notifications with app password
Replaced the Gmail API (OAuth2) notification delivery with plain
SMTP using a Gmail app password. Much simpler setup — no Google
Cloud project, no OAuth2 flow, no extra dependencies.

- Rewrote _send_gmail() to use smtplib with smtp.gmail.com:587 TLS
- Added stonks-gmail-secrets to Helm chart (GMAIL_SENDER,
  GMAIL_RECIPIENT, GMAIL_APP_PASSWORD)
- Added gmail secret to trading-engine deployment
- Updated runmefirst.sh to read gmail.app from kube dir
- Sender/recipient: celes@celestium.life
2026-04-16 02:37:40 +00:00
Celes Renata 9aae57f3e1 docs: rewrite README and runbook for current platform state
README: updated architecture diagram, three signal layers, tracked
universe, autonomous trading engine, global news interpolation,
competitive intelligence, paper trading, notification service,
updated services table, project structure, deployment, endpoints.

Runbook: updated service overview, deployment via runmefirst.sh,
secrets management (keys in kube dir not repo), backup/restore
scripts, trading engine operations, signal layer toggles, database
nuke & rebuild, monitoring, CI/CD, removed hardcoded secrets.
2026-04-16 02:06:18 +00:00
Celes Renata e652a62dbc fix: Trading Controls page field mapping for macro/competitive status
The API returns macro_enabled/competitive_enabled but the TypeScript
interfaces expected 'enabled'. The toggles always showed disabled.
Now handles both field names with fallback.
2026-04-16 01:46:13 +00:00
Celes Renata 88c9f50371 fix: treat 404 on Alpaca positions endpoint as empty result
Alpaca returns 404 when you don't hold a position in a ticker.
The ingestion worker was logging this as an error and incrementing
the failure count. Now returns an empty items list instead, since
'no position' is a valid state, not an error.
2026-04-16 01:33:35 +00:00
Celes Renata 00ea917fc0 fix: add broker secrets to ingestion worker deployment
The ingestion worker creates an AlpacaBrokerAdapter but the pod
didn't have BROKER_API_KEY/BROKER_API_SECRET env vars, causing
401 Unauthorized on every broker source fetch. Added
stonks-broker-secrets to the ingestion service's secrets list.
2026-04-16 01:25:54 +00:00
Celes Renata 36c92196d2 fix: remove unused Legend import from Dashboards (TS strict) 2026-04-16 01:11:23 +00:00
Celes Renata 949324dc89 feat: SQL Explorer with PostgreSQL schema browser and pre-built queries
The SQL Explorer was querying Trino which has zero tables. Rewrote to
use PostgreSQL directly:

Backend:
- GET /api/analytics/pg-schema: returns all public tables with column
  names, types, and nullability from information_schema
- POST /api/analytics/pg-query: read-only SQL execution against
  PostgreSQL with SELECT-only enforcement, auto LIMIT, and descriptive
  error messages for syntax/table/query errors

Frontend:
- Schema browser shows all PostgreSQL tables with columns and types
- Click a table name → generates SELECT * FROM table LIMIT 100
- Pre-built Queries section with 12 seeded queries covering companies,
  recommendations, trends, market prices, documents, global events,
  trading decisions, ingestion health, reserve pool, sector exposure
- User-saved queries shown separately with delete buttons
- Chart builder, Monaco editor, and save functionality preserved

Migration 021: seeds 12 pre-built saved queries
2026-04-16 01:06:49 +00:00
Celes Renata 55512ca5a8 fix: rewrite dashboards to use PostgreSQL API instead of empty Trino lakehouse
The Trino/Iceberg lakehouse has zero tables, so all Trino-backed
dashboards showed 'No data available'. Rewrote all four to use
existing PostgreSQL-backed API endpoints:

- Sentiment Heatmap: useTrends + useCompanies → sector and ticker
  trend strength bar charts (30k trend_windows in DB)
- Prediction Accuracy: useRecommendations → confidence distribution
  and action distribution charts (30k recommendations in DB)
- Paper PnL: useTradingMetrics + useTradingMetricsHistory → equity
  curve, daily returns, win/loss stats from trading engine
- Model Quality: useModelPerformance + useModelFailures → success
  rate, latency, retries, and failure table from ops API

Removed unused Trino query function and ScatterChart imports.
2026-04-16 00:58:18 +00:00
Celes Renata b5c0c6d7c9 fix: aggregate ingestion throughput chart by time bucket
The throughput API returns one row per source_type per time bucket,
but the chart was mapping each row as a separate bar. With 5 source
types × 24 hours, the bars were tiny and overlapping. Now aggregates
completed/failed/items across source types per time bucket so the
chart shows meaningful totals.
2026-04-16 00:52:29 +00:00