Commit Graph

100 Commits

Author SHA1 Message Date
Celes Renata c501ccea40 fix: default model to qwen3.5:9b + improve event classifier prompt
- Migration 026 and OllamaConfig now default to qwen3.5:9b instead of
  llama3.1:8b. Existing deployments keep their current model (qwen3.5:9b-fast)
  since the migration uses WHERE NOT EXISTS on slug.

- Event classifier system prompt expanded with macro-vs-company filtering:
  explicitly instructs the model to NOT classify single-company news
  (lawsuits, earnings, management changes, debt crises) as macro events.
  Sets severity=low and confidence<0.3 for company-specific articles.
  Reserves 'critical' severity for multi-country/global market events.
  Prevents over-tagging event_types by requiring direct description.

- Updated test_system_prompt_is_concise threshold to accommodate the
  expanded prompt (300 → 1000 chars).
2026-04-17 02:53:38 +00:00
Celes Renata 90614dd7bb feat: paper trading capital controls — add, withdraw, and full reset
Three distinct capital operations on the Trading Controls page:

- Set Capital: overwrites pool balances to a new amount (existing)
- Add/Withdraw: adjusts active pool by a delta without touching
  positions, orders, or history. Validates sufficient balance for
  withdrawals. Logged to reserve_pool_ledger as manual_adjustment.
- Reset Everything: nuclear option — deletes all positions, orders,
  trading decisions, stop levels, snapshots, backtests, notifications,
  and circuit breaker events, then resets capital fresh. Red button
  with double-confirmation dialog.

Backend: POST /api/trading/capital/adjust and POST /api/trading/reset
Frontend: CapitalCard rebuilt with three sections and confirmation UIs
2026-04-17 02:23:26 +00:00
Celes Renata 45752b9a29 feat: AI Agents management page with per-agent performance tracking
New Agents tab in the sidebar (Ops group) for viewing, editing, and
creating AI agent configurations:

Database (migration 026):
- ai_agents table: editable configs for each LLM agent (model, prompts,
  temperature, tokens, retries). source='system' for built-in,
  source='user' for custom. Seeds 3 system agents (Document Extractor,
  Event Classifier, Thesis Rewriter) using WHERE NOT EXISTS to never
  overwrite user edits across reinstalls.
- agent_performance_log table: per-invocation metrics (duration,
  confidence, retries, tokens, errors) linked to agent config.

API endpoints:
- GET/POST /api/agents — list and create agents
- GET/PUT/DELETE /api/agents/{id} — view, edit, delete (system agents
  can be edited but not deleted)
- GET /api/agents/{id}/performance — aggregated metrics (success rate,
  avg/p95 latency, confidence, token usage)
- GET /api/agents/{id}/performance/history — hourly time series

Frontend:
- AgentsPage with sidebar list + detail panel
- Agent detail: config display, system prompt viewer, performance
  dashboard with metrics cards and time-series chart
- Edit form: all config fields editable including system prompt,
  model, temperature, tokens, retries
- Create form: new user-defined agents with auto-slug generation
- System agents show blue badge, user agents show green badge
2026-04-17 01:24:35 +00:00
Celes Renata 2360c501e4 feat: intraday hourly price bars via Polygon range endpoint
- New 'intraday_bars' endpoint in PolygonMarketAdapter: fetches hourly
  bars for today using range_bars URL with timespan=hour, sort=asc
- Scheduler expands intraday_bars global source into per-ticker jobs
  for all active companies (every 15 minutes via polling_interval)
- Migration 025 inserts the intraday source with 900s cadence
- Frontend price matching uses closest-timestamp instead of date-string
  matching, with 2h tolerance for intraday and 36h for daily windows
- Bumped market price fetch limit to 200 for intraday granularity
2026-04-17 01:13:24 +00:00
Celes Renata c4206b3f4c feat: overlay stock price on trend charts with right Y axis
- New GET /api/market/prices/{ticker} endpoint serving OHLCV data from
  market_snapshots, deduped by bar_timestamp
- New useMarketPrices hook in frontend
- Trend chart now shows price (purple line) on a right Y axis ($)
  alongside trend metrics (%) on the left Y axis
- Custom tooltip formats price as dollars, metrics as percentages
- Price line uses connectNulls for days with missing bar data
2026-04-17 01:09:36 +00:00
Celes Renata 7c589353f8 fix: blank company charts + competitor GUIDs instead of tickers
Trend charts blank:
- trend_windows uses upsert (1 row per ticker/window), so charts had
  at most 1 data point. Added trend_history table (migration 024) that
  appends every snapshot. New /api/trends/history endpoint serves the
  time series. Frontend now uses useTrendHistory for charts and
  useTrends for the latest summary card.

Competitor GUIDs:
- list_competitors query returned raw company_b_id UUIDs without
  joining companies table. Added LEFT JOIN with CASE to resolve the
  other company's ticker and legal_name. Updated Pydantic model to
  include enriched fields. Frontend fallback changed from truncated
  UUID to ticker/legal_name/Unknown.
2026-04-17 00:42:55 +00:00
Celes Renata f2d8744a4f fix: backtest submission shows no results — 4 bugs fixed
- ID mismatch: API generated a throwaway UUID while BacktestReplay
  generated its own internally. Frontend polled with wrong ID and
  never found the DB row. Now pre-generate ID in endpoint and pass
  it to BacktestReplay.
- Field name: API returned 'backtest_id' but frontend read 'data.id'.
  Unified to 'id' everywhere.
- No polling: useBacktestResult fired once and never refreshed.
  Added refetchInterval that polls every 2s while status is running.
- Response shape: GET endpoint nested results under 'result' object
  but frontend expected flat fields. Flattened response to match
  BacktestResult type.
- Added running/failed/completed status indicators in BacktestPanel.
2026-04-17 00:31:17 +00:00
Celes Renata f11aa0a1ee fix: deduplicate recommendations and widen position sizing range
- Add dedup check in recommendation worker: skip generation when latest
  rec for same ticker+window has identical action/mode/confidence
- Widen position sizing range (1-10% portfolio, 0.3-2% max loss) and
  factor in trend strength + evidence count for differentiated sizing
- API returns only latest recommendation per ticker by default (DISTINCT ON)
  to eliminate duplicate rows in the frontend list view
2026-04-17 00:15:32 +00:00
Celes Renata 29f46d387c fix: 6 buy/sell logic bugs — sells check trading window, persist audit trail, dedup after position check, no duplicate buys, fix stop-level insert, profit-taking respects market hours 2026-04-17 00:07:50 +00:00
Celes Renata 1246b3868b fix: use current_price not avg_entry_price for invested calc — prevents margin-inflated numbers showing $0 available 2026-04-17 00:04:12 +00:00
Celes Renata f57167ce4d fix: only poll recommendations from last 2 hours (not 24h), persist snapshots during market hours for performance tab 2026-04-16 23:53:54 +00:00
Celes Renata 18eb150c75 fix: confidence formula now uses unique doc count + signal agreement instead of raw signal count — prevents 99.9% inflation 2026-04-16 23:48:05 +00:00
Celes Renata 1a5fb2e36a fix: recovery sweep touches updated_at after re-enqueue to prevent duplicate flooding every 5 min 2026-04-16 19:13:10 +00:00
Celes Renata 693d9e0d60 fix: reduce LLM timeouts — truncate docs to 8k/6k chars, cut num_predict 16k→4k, tighten prompts, trim anti-hallucination rules 2026-04-16 18:56:11 +00:00
Celes Renata 60cfb7618e fix: recovery sweep skips docs that already have global_events — prevents re-enqueue loop 2026-04-16 18:27:21 +00:00
Celes Renata 1043710b6d fix: track last_published_at per source to avoid re-fetching same articles — applies to both news_api and macro_news 2026-04-16 18:12:12 +00:00
Celes Renata 513310abba fix: stop tagging all macro_news articles as macro_event — default to article, let extractor reclassify. Also reduced fetch limit to 20 and cadence to 30min 2026-04-16 18:09:50 +00:00
Celes Renata f83577480f fix: alternate extractor between macro and extraction queues (1:2 ratio) to prevent starvation 2026-04-16 17:45:25 +00:00
Celes Renata 2440cddd37 feat: add comprehensive table retention cleanup to scheduler — 10 tables with per-table retention windows 2026-04-16 15:56:50 +00:00
Celes Renata 63287903d0 feat: wire up stop levels, circuit breaker daily loss, profit-taking, real portfolio/decisions/history endpoints 2026-04-16 15:52:46 +00:00
Celes Renata 1329df0bbf feat: sell execution, correlation matrix from market data, US market holiday awareness
- Sell path: looks up existing position, sells full quantity, returns proceeds to pool
- Correlation matrix: computed from 30-day market_snapshots on startup + every 5min
- Holidays: 10 major US market holidays for 2026 checked in trading window functions
2026-04-16 15:36:49 +00:00
Celes Renata 2e77cf32fd fix: critical — track capital properly: load invested positions on startup, deduct on act, sync every 5min 2026-04-16 15:29:28 +00:00
Celes Renata 9a8d36068a fix: convert Decimal to float in API responses instead of string — fixes positions page crash 2026-04-16 15:25:40 +00:00
Celes Renata 354c3d484a fix: fetch current prices from market_snapshots before evaluating recommendations — fixes 'Invalid current price' skip 2026-04-16 15:17:49 +00:00
Celes Renata 2a6aac47a6 fix: add decision logging to trading engine, flushed 103k stale dedup keys 2026-04-16 15:12:58 +00:00
Celes Renata c114e77b1c fix: limit recommendation poll to 50 per cycle to prevent 85k-rec processing stall, add poll logging 2026-04-16 15:05:26 +00:00
Celes Renata 136b149000 fix: add logging config to trading engine, add /api/trading/debug diagnostic endpoint 2026-04-16 14:55:42 +00:00
Celes Renata 6bab199159 fix: trend_windows now upserts instead of accumulating (7.5GB→4MB), add competitive signal retention cleanup 2026-04-16 14:32:24 +00:00
Celes Renata 58a8726306 feat: add paper trading capital controls — API endpoint + UI with presets, fix status/metrics to read real state, fix migration duplicates 2026-04-16 14:06:30 +00:00
Celes Renata 14e411daf9 fix: trading status and metrics endpoints now read real portfolio state instead of hardcoded zeros 2026-04-16 14:02:38 +00:00
Celes Renata 5cc64498c0 fix: skip already-propagated docs in aggregation, limit to last hour + 10 docs instead of 50 2026-04-16 09:35:11 +00:00
Celes Renata f0887afd9b fix: cap competitive signals to 500 most recent per window to prevent 67k row aggregation bottleneck 2026-04-16 09:24:31 +00:00
Celes Renata 58a05ca322 feat: add SSE stream for live pipeline status, add all 10 queues + DLQs, configure nginx for SSE 2026-04-16 08:15:44 +00:00
Celes Renata 87579d68da fix: add stale document recovery sweep to scheduler, re-enqueues orphaned parsed docs every 5 min 2026-04-16 07:59:30 +00:00
Celes Renata cdc825619e feat: add live queue depths to pipeline health API and dashboard 2026-04-16 07:49:07 +00:00
Celes Renata 0ee7f26633 feat: raise market_api rate to 20/min, add global Polygon cap at 45/min, add rate-limit API + watchlist warning 2026-04-16 07:26:10 +00:00
Celes Renata 1107d34027 fix: SQL Explorer handles comments and shows descriptive errors
- Strip SQL comments (-- and /* */) before checking for SELECT,
  so queries with leading comments don't get rejected
- Show the actual error detail from the API response instead of
  generic 'API error 400' in the SQL Explorer UI
2026-04-16 05:25:45 +00:00
Celes Renata c4666c071b feat: wire Gmail SMTP notifications with app password
Replaced the Gmail API (OAuth2) notification delivery with plain
SMTP using a Gmail app password. Much simpler setup — no Google
Cloud project, no OAuth2 flow, no extra dependencies.

- Rewrote _send_gmail() to use smtplib with smtp.gmail.com:587 TLS
- Added stonks-gmail-secrets to Helm chart (GMAIL_SENDER,
  GMAIL_RECIPIENT, GMAIL_APP_PASSWORD)
- Added gmail secret to trading-engine deployment
- Updated runmefirst.sh to read gmail.app from kube dir
- Sender/recipient: celes@celestium.life
2026-04-16 02:37:40 +00:00
Celes Renata 88c9f50371 fix: treat 404 on Alpaca positions endpoint as empty result
Alpaca returns 404 when you don't hold a position in a ticker.
The ingestion worker was logging this as an error and incrementing
the failure count. Now returns an empty items list instead, since
'no position' is a valid state, not an error.
2026-04-16 01:33:35 +00:00
Celes Renata 949324dc89 feat: SQL Explorer with PostgreSQL schema browser and pre-built queries
The SQL Explorer was querying Trino which has zero tables. Rewrote to
use PostgreSQL directly:

Backend:
- GET /api/analytics/pg-schema: returns all public tables with column
  names, types, and nullability from information_schema
- POST /api/analytics/pg-query: read-only SQL execution against
  PostgreSQL with SELECT-only enforcement, auto LIMIT, and descriptive
  error messages for syntax/table/query errors

Frontend:
- Schema browser shows all PostgreSQL tables with columns and types
- Click a table name → generates SELECT * FROM table LIMIT 100
- Pre-built Queries section with 12 seeded queries covering companies,
  recommendations, trends, market prices, documents, global events,
  trading decisions, ingestion health, reserve pool, sector exposure
- User-saved queries shown separately with delete buttons
- Chart builder, Monaco editor, and save functionality preserved

Migration 021: seeds 12 pre-built saved queries
2026-04-16 01:06:49 +00:00
Celes Renata 6eda988e3b fix: strip /v2 suffix from broker base URL to prevent doubled path
The alpaca.url config file contains https://paper-api.alpaca.markets/v2
but the adapter code also prepends /v2/ to all paths, resulting in
/v2/v2/positions which returns 404. Now strips trailing /v2 or /v1
from the configured base URL since the adapter manages API versioning.

This was causing 1,017 consecutive broker sync failures.
2026-04-16 00:45:19 +00:00
Celes Renata 88c2bc84a1 feat: upgrade paper trading to $100k moderate tier
Paper money has no downside — bigger capital exposes more model
behavior: position sizing, diversification, sector exposure,
correlation checks, circuit breakers, reserve pool siphoning,
and risk tier auto-adjustment all become meaningful.

- risk_tier: conservative → moderate (min_confidence 0.55)
- absolute_position_cap: $25 → $10,000
- max_open_positions: 5 → 10
- initial portfolio value: $500 → $100,000
- Updated migration 019, Helm values, and engine default
2026-04-16 00:37:35 +00:00
Celes Renata d21110b3d2 fix: backtest skips duplicate ticker positions to prevent overwrite
When multiple recommendations for the same ticker produce 'act'
decisions, the second one would overwrite the first in
simulated_positions, losing the first position's value and causing
incorrect portfolio value calculations. Now skips if already holding.
2026-04-16 00:28:59 +00:00
Celes Renata 2eaf6dc025 fix: upgrade backtest day summary to WARNING level for visibility 2026-04-16 00:21:46 +00:00
Celes Renata bad7e02e53 fix: add first-skip-reason logging to backtest replay for debugging 2026-04-16 00:16:32 +00:00
Celes Renata 4634f1f3fc fix: remove unused skip_count variable (lint) 2026-04-16 00:07:29 +00:00
Celes Renata ff5055ee4e fix: backtest replay field mapping and logging
- Map DB 'id' field to 'recommendation_id' for evaluate_recommendation()
- Ensure confidence is cast to float (asyncpg may return Decimal)
- Add per-day logging showing rec count, act/skip, positions, pool balance
- Helps diagnose why backtests produce 0 trades
2026-04-15 22:55:26 +00:00
Celes Renata 4501bbebd4 feat: add Polygon grouped daily endpoint for broad market data
Two tiers of market data:
1. Per-ticker prev bars (existing 50 sources, 15-min cadence) for
   watchlist detail — trading decisions, stop-loss, position sizing
2. Grouped daily (new single source, once per day) for broad market
   context — correlation analysis, sector rotation, competitive intel

Changes:
- Add grouped_daily endpoint to PolygonMarketAdapter with auto date
  calculation (previous trading day, skip weekends)
- Add fetch_global_market_sources() to scheduler for sources without
  company_id, scheduled once daily (86400s cadence)
- Update _persist_market_items to use item-level ticker from T field
  and look up company_id dynamically for grouped daily bars
- Migration 020: make company_id nullable on sources and
  market_snapshots tables, add grouped daily source row
- Fix backtest replay to query market_snapshots data->>'c' for prices
2026-04-15 22:38:18 +00:00
Celes Renata ea6c2b3f54 fix: market data rate limiting and backtest price lookup
- Increase market_api polling cadence from 60s to 900s (15 min).
  The prev-day bar endpoint returns the same data all day, so polling
  every minute wastes API quota. 50 tickers at 15-min cadence = ~3.3
  req/min, well within the 5/min rate limit.
- Reduce market_api rate limit from 30/min to 5/min to match.
- Fix backtest replay to query market_snapshots with data->>'c' for
  close prices instead of nonexistent market_data.close_price column.
- Enrich backtest recommendations with prices from market_snapshots
  and sectors from companies table.
2026-04-15 22:19:44 +00:00
Celes Renata 69eb366bf9 fix: backtest simulation uses ET timezone for trading window
The simulated timestamp was 10:00 UTC (6:00 AM ET) which is outside
the trading window. Changed to 11:00 AM ET so backtested decisions
actually pass the trading window check.
2026-04-15 21:48:32 +00:00