179 Commits

Include renames

Author	SHA1	Message	Date
Celes Renata	a72f336ad1	feat: Intelligence Pipeline v3 — full implementation ... Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.	2026-07-13 02:14:59 +00:00
Celes Renata	ed474c521a	fix: update document status after macro classification ... _process_macro_classification never updated document status, leaving macro_event docs stuck in parsed forever. Now marks them extracted on success or extraction_failed on error.	2026-07-10 22:09:02 +00:00
Celes Renata	d9fbd5b660	fix: handle null fields from NuExtract3 in extraction normalization ... - Replace setdefault() with explicit None checks in _normalize_extraction_data() - Coerce null top-level fields (summary, novelty_score, confidence) to defaults - Coerce null company fields (ticker, impact_score, impact_horizon, etc.) to defaults - Filter out company entries with empty ticker after normalization - Prevents schema validation failures when model returns null for required fields	2026-07-10 20:38:50 +00:00
Celes Renata	ca712ad4a0	fix: pipeline health — stuck docs, price fallback, sentiment normalization, signal-engine scale, quality gate ... - Scheduler: lower stale threshold 240→30 min, batch limit 100→500, TTL 14400→3600 - Prediction snapshot: add 24h market_snapshots time-window fallback - Aggregation: add normalize_impact_scores() z-score normalization - Helm: signal-engine replicas → 0 (idle when dual pipeline disabled) - Quality gate: max_snapshot_age_hours 24→48 - Add backfill script for NULL price_at_prediction snapshots - Add PBT bug condition and preservation tests (14 tests)	2026-07-10 20:16:01 +00:00
Celes Renata	14a9b4fcc1	fix: ops pipeline — rate limit, validation cycle, snapshots, config query, rejection reason ... - Polygon rate limit env-configurable, default 5 (free tier) - v3_engine_enabled reads from JSONB config column correctly - Validation cycle (outcome eval + metrics) wired into scheduler hourly - Daily portfolio/risk snapshots after 16:30 ET with idempotency - Prediction snapshot price fallback from positions table - Order rejection_reason + rejected_at persisted on insert - Lake-publisher scaled to 0 replicas (redundant) - Test fix: isolate per-type rate limit tests from Polygon global	2026-07-03 08:29:24 +00:00
Celes Renata	b4bf0f2361	feat: math core v3 engine upgrade	2026-06-27 12:21:41 +00:00
Celes Renata	f468e30af0	feat: implement dual-pipeline signal engine service ... New service at services/signal_engine/ implementing concurrent heuristic (deterministic scoring) and probabilistic (Bayesian inference) pipelines that evaluate technical signals across 6 timeframes (M30-M) and produce independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick. Components: - Input Normalizer: multi-source data assembly with sentinel fallbacks - Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave - Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors - Hard Filter Engine: macro_bias, valuation, earnings proximity gating - Heuristic Pipeline: S_total scoring with confidence-gated verdicts - Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy gating, EV_R calculation, and signal correlation penalty - Exit Engine: stop-loss, targets, trailing ATR-based stops - Delta Analyzer: pipeline agreement tracking with rolling Redis metrics - Output Formatter: SignalOutput contract + Recommendation schema mapping - Worker orchestrator: concurrent pipelines with failure isolation - Main entry point: queue polling with fail-safe config loading Infrastructure: - Migration 039: signal_engine_outputs table with 3 indexes - Helm chart: signalEngine service entry (processing tier) - Redis key: QUEUE_SIGNAL_ENGINE constant Tests: 390 tests (unit + property-based) covering all components Config: dual_pipeline_enabled=false by default (safe rollout)	2026-05-02 07:32:26 +00:00
Celes Renata	bc077bfcc8	feat: trading feedback engine — periodic performance reports with AI summarization ... - Migration 038: trading_reports table + report-summarizer agent seed - 6 reporting modules: models, collector, sections, validator, summarizer, generator - API endpoints: GET /api/reports (paginated, filterable), GET /api/reports/{id} - Frontend hooks: useReports, useReport with TanStack Query - Scheduler: daily (after 16:30 ET) and weekly (Saturday) report triggers - Redis queue consumer for async report generation with retry/dedup - 5 property-based tests (chunking, serialization, validation, accuracy, deltas) - 109 unit/integration tests across all modules - 6 frontend hook tests with MSW mocks	2026-05-01 22:13:09 +00:00
Celes Renata	d9110d03a6	fix: use di.document_id instead of dir.document_id in evidence query ... document_impact_records has intelligence_id, not document_id. The document_id lives on document_intelligence.	2026-05-01 04:06:55 +00:00
Celes Renata	76f6bd5677	fix: quote reserved keyword 'window' in prediction_snapshots SQL	2026-05-01 03:40:48 +00:00
Celes Renata	7fcc8a6c07	feat: model validation, calibration, and signal quality layer ... - Migration 035: prediction_snapshots, prediction_outcomes, signal_evidence_links, model_metric_snapshots tables + SQL views - Prediction snapshot writer with canonical evidence keys, duplicate detection, contribution scores - Outcome evaluator across 5 horizons (1h, 6h, 1d, 7d, 30d) - Metrics engine: ECE, Brier score, IC, Rank IC, benchmark comparison - Attribution engine: per-source, per-catalyst, per-layer performance - Calibration engine: Bayesian shrinkage source reliability - Quality gate for live trading eligibility with configurable thresholds - 7 new /api/validation/* endpoints - Upgraded OpsModel dashboard with validation tab - Enhanced recommendation display with calibration context - Backtest replay validation mode - 86 Python tests (unit + property-based), 179 frontend tests passing	2026-05-01 03:04:58 +00:00
Celes Renata	2f2ea65fb4	fix: keep Alpaca price as primary (includes after-hours), add polygon_price field ... Alpaca's current_price reflects extended hours trading which is more current than Polygon's regular session close. Keep it as the display price. Add polygon_price as a reference field in the API response.	2026-04-30 22:35:22 +00:00
Celes Renata	9a60ce127b	fix: overlay Polygon prices on positions instead of stale Alpaca prices ... Alpaca paper trading returns inaccurate current_price values. The positions endpoint now uses the latest Polygon close from market_snapshots and recomputes unrealized P&L from that.	2026-04-30 22:32:21 +00:00
Celes Renata	414f476620	fix: auto-backfill market data on startup + fix trading dedup bug ... 1. Add backfill-market-data init container to scheduler deployment. On vanilla start (< 50 bars in market_snapshots), fetches 90 days of daily OHLCV from Polygon directly via asyncpg + httpx. 2. Fix trading engine dedup key placement. Previously the Redis dedup key was set BEFORE evaluate_recommendation(), so recs skipped as outside_trading_window were permanently deduped and never retried when the market opened. Now the dedup key is only set AFTER the decision for non-retryable outcomes (act, confidence_too_low, etc). outside_trading_window skips are retryable.	2026-04-30 14:08:54 +00:00
Celes Renata	bddaf44ffc	feat: 90-day price range Y-axis scaling with breakthrough annotations ... Backend: - GET /api/market/prices/{ticker} now returns { bars, range_90d } with 90-day low/high computed from market_snapshots - POST /api/market/backfill/{ticker} fetches 90 days of daily bars from Polygon and inserts missing bars into market_snapshots - POST /api/market/backfill-all does the same for all active tickers Frontend: - Right Y-axis domain scaled to 90-day min/max (with 3% padding) - Green dashed reference line at 90-day high - Red dashed reference line at 90-day low - Labels show exact price on each reference line - Default limit bumped to 200 bars	2026-04-30 06:09:05 +00:00
Celes Renata	5209cc522e	feat: enhanced paper trading reset with capital and reserve controls ... - Add initial capital input (toggle between broker balance or custom amount) - Add reserve/active pool split slider (0-50%, default 20%) - Backend accepts reserve_pct in reset request body - Note in UI that Alpaca balance reset requires Alpaca dashboard - Confirmation dialog shows exact capital and split being applied	2026-04-30 05:49:38 +00:00
Celes Renata	13f863ef30	feat: fetch 15-minute bars instead of hourly for intraday prices ... Changed intraday_bars default from 1-hour bars to 15-minute bars. This gives ~26 price points per trading day per ticker (6.5h market × 4 bars/hour) instead of ~7 hourly bars. Limit raised to 100 to accommodate the higher bar count.	2026-04-29 23:01:11 +00:00
Celes Renata	2538da3f1e	fix: switch market data sources from prev_bars to intraday_bars ... The market_api sources were configured with endpoint='prev_bars' which only fetches a single previous-day bar per ticker. Changed to 'intraday_bars' which fetches hourly bars for today from Polygon's /v2/aggs/ticker/{ticker}/range/1/hour/{today}/{today} endpoint. Updated: seed script, beta DB (50 sources), production DB (50 sources). This gives ~7-8 hourly price bars per trading day per ticker instead of 1 daily bar.	2026-04-29 22:07:47 +00:00
Celes Renata	cb3eb230d6	fix: fetch trend history per-window and run aggregation 24/7 ... Two fixes for missing intraday data: 1. Frontend: lifted selectedWindow state to page level so useTrendHistory passes window param to the API. Previously fetched all windows with limit=500 which exhausted the limit before reaching recent intraday data. Now fetches only the selected window's data. 2. Scheduler: removed market-hours-only restriction from periodic aggregation. Runs every 15 minutes 24/7 so intraday data is always populated for backtesting regardless of market state.	2026-04-29 21:11:46 +00:00
Celes Renata	6f54fd07fa	feat: periodic aggregation every 15 minutes during market hours ... The aggregation engine only ran when new documents were ingested, leaving intraday trend data stale for long periods. Now the scheduler enqueues all 50 tickers for re-aggregation every ~15 minutes during US market hours (Mon-Fri, 6:30 AM - 1:30 PM PT). This ensures continuous intraday trend updates based on existing signals and market price changes.	2026-04-29 18:27:49 +00:00
Celes Renata	6880f11c26	fix: add /no_think inline tag to disable Qwen3 thinking mode ... chat_template_kwargs isn't being respected by the vLLM deployment. Qwen3 models support /no_think as an inline suffix in the user message to disable thinking mode. This is the most reliable method across all serving backends (vLLM, Ollama, SGLang).	2026-04-29 16:11:44 +00:00
Celes Renata	eead4f1381	fix: disable thinking mode on vLLM path with chat_template_kwargs ... The thesis rewriter uses vLLM (not Ollama) in production. The previous fix only added think=False to the Ollama payload. For vLLM's OpenAI-compatible API with Qwen3 models, thinking mode is disabled via chat_template_kwargs: {enable_thinking: false} in the request body.	2026-04-29 16:04:04 +00:00
Celes Renata	007189c0a5	fix: handle plain-text thinking blocks and disable think mode ... The model outputs 'Thinking Process:' as plain text (not in <think> tags). Updated _strip_thinking_block to handle both XML tags and plain-text reasoning patterns. Also: - Added rule 7 to system prompt: 'Do NOT show your thinking process' - Set think=False in Ollama payload to disable Qwen3 thinking mode - Added fallback regex to extract thesis from after thinking blocks	2026-04-29 15:50:49 +00:00
Celes Renata	f9ee1532dc	fix: strip <think> reasoning blocks from thesis LLM output ... Qwen3.5 in thinking mode emits <think>...</think> chain-of-thought before the actual response. The thesis rewriter was returning the raw output including the entire reasoning block. Now strips thinking tags from both Ollama and vLLM response paths.	2026-04-29 15:25:04 +00:00
Celes Renata	bb40a3cb8e	fix: position sync now reconciles — removes positions broker no longer holds ... The sync_positions loop only upserted positions from Alpaca but never deleted DB rows for positions that were closed/liquidated on the broker side. After a paper reset, the next sync would not remove the stale positions because they simply weren't in Alpaca's response anymore. Now performs full reconciliation: after upserting what Alpaca reports, deletes any DB positions for the account that Alpaca no longer holds.	2026-04-29 12:02:57 +00:00
Celes Renata	4e010bc048	feat: signal math upgrade — probabilistic, regime-aware scoring pipeline ... Implement full probabilistic signal processing pipeline gated behind probabilistic_scoring_enabled feature flag in risk_configs: - Bayesian log-likelihood accumulator with Beta posterior and entropy - Regime detector (trend-following, panic, mean-reversion, uncertainty) - Source accuracy tracker with per-source historical prediction accuracy - Sigmoid confidence gate replacing binary gate - Information gain surprise weighting for rare events - Adaptive recency decay with event-specific half-lives - Regime multiplier replacing market context multiplier - Weighted disagreement entropy for contradiction detection - Multiplicative macro exposure with conditional integration - Graph-distance attenuated competitive signal propagation - Exponentially weighted momentum with volatility scaling - Expected value recommendation gate All changes backward-compatible: flag=false preserves exact current behavior. New outputs stored in existing JSONB columns (no schema changes except source_accuracy table via migration 034). Tests: 26 property-based tests (14 correctness properties), 99 unit tests, 1789 total tests passing with zero regressions.	2026-04-29 11:41:48 +00:00
Celes Renata	8c3c1aab43	fix: pipeline stop now halts all workers and flushes queues ... Workers (ingestion, parser, extractor, aggregation, recommendation, broker, lake-publisher) now check the pipeline:enabled Redis flag on each loop iteration and sleep when disabled. The toggle endpoint flushes all pipeline queues on disable so queued jobs don't resume when workers eventually check. Broker/trading queues are excluded from flush to avoid dropping in-flight orders.	2026-04-29 07:59:35 +00:00
Celes Renata	f264e924f0	fix: thesis rewriter now routes to vLLM when provider is vllm ... - thesis_llm.py: add _call_vllm_thesis() using /v1/chat/completions - thesis_llm.py: check resolved model_provider and route accordingly - values.yaml: set OLLAMA_BASE_URL to http://10.1.1.12:2701	2026-04-29 05:42:10 +00:00
Celes Renata	b38fb24f14	fix: ensure production uses DB-configured model/provider from UI ... - Migration 026: update seed defaults from ollama to vllm/AxionML - Migration 031: fix existing rows still on old ollama defaults - Helm values: set OLLAMA_BASE_URL to cluster ollama endpoint (was empty) - Extractor: guard against switching to ollama when base_url is empty - OllamaClient: validate base_url on construction to fail fast	2026-04-29 04:33:21 +00:00
Celes Renata	49bff9de50	fix: delete stale trend_evidence before inserting to prevent duplicate accumulation	2026-04-28 19:45:15 +00:00
Celes Renata	27b84fcd2e	fix: return open_position_count in trading status for dashboard display	2026-04-28 19:27:40 +00:00
Celes Renata	7e8d518946	feat: add max open positions and position cap controls to trading dashboard	2026-04-28 19:16:50 +00:00
Celes Renata	c188677330	fix: route macro_news documents as macro_event so global event classification runs	2026-04-28 16:18:02 +00:00
Celes Renata	226d799eb2	feat: auto-clamp buy orders to fit within position limits instead of hard-rejecting	2026-04-28 14:20:44 +00:00
Celes Renata	e360b66c3e	fix: beta trading pipeline — max_tokens default, approval re-enqueue, credentials ... - Migration 031: change ai_agents/agent_variants max_tokens default from 32768 to 4096 (32768 exceeds vLLM context window, causing HTTP 400 on every extraction) - API: re-enqueue approved orders to broker queue — previously approved orders sat in DB with nothing to execute them - values-beta: enable TRADING_ENABLED, update Alpaca paper keys	2026-04-28 14:13:58 +00:00
Celes Renata	0437943863	fix: reduce vLLM default max_tokens to 4096, update model to AxionML/Qwen3.5-9B-NVFP4 ... The model's max_model_len is 16384 — requesting 32768 output tokens caused HTTP 400 from vLLM. 4096 is a safe default for extraction output.	2026-04-23 19:49:34 +00:00
Celes Renata	f7ae34ef3b	fix: add extract() method to VLLMClient for extraction pipeline compatibility	2026-04-23 19:32:33 +00:00
Celes Renata	117b693b19	feat: add remote vLLM support with provider abstraction layer ... - LLMClient Protocol for provider-agnostic inference - VLLMClient for OpenAI-compatible /v1/chat/completions API - LLM client factory with provider routing (ollama/vllm) - VLLMConfig with VLLM_* environment variable loading - Updated extractor worker with health check and provider switching - Updated event classifier to use LLMClient protocol - Helm values for vLLM configuration - 18 unit tests + 6 property-based tests - Full backward compatibility preserved	2026-04-23 08:17:23 +00:00
Celes Renata	f251c53f92	fix: risk engine blocking sell orders on over-concentrated positions ... Two bugs: (1) trading engine omitted estimated_value from sell order jobs, causing risk engine to compute 0 reduction; (2) risk engine applied position size limits to sells, trapping users in positions they couldn't exit. Sells now always pass position value/pct checks.	2026-04-22 02:07:24 +00:00
Celes Renata	3b49aa2fa2	fix: risk engine now allows sells on over-concentrated positions	2026-04-21 20:25:02 +00:00
Celes Renata	5d0635a291	feat: beta deploys all services with pipeline toggle defaulting to OFF ... - pipelineEnabled: true in beta so all pods run (Kargo happy) - PIPELINE_DEFAULT_OFF=true in beta config — scheduler initializes the Redis toggle to OFF on first boot - Shared Ollama (10.1.1.12:2701) between beta and paper - Flip pipeline ON from the UI when testing, OFF when done - Optimistic UI update for the toggle button	2026-04-21 03:54:00 +00:00
Celes Renata	be526ae614	feat: pipeline on/off toggle with per-stage Helm control ... - Added pipelineEnabled flag to Helm values (default: true) - Worker services (scheduler, ingestion, parser, extractor, aggregation, recommendation, broker-adapter, lake-publisher) scale to 0 when disabled - API services always run regardless of toggle - Redis-based runtime toggle: POST /api/ops/pipeline/toggle - Scheduler checks the flag before each cycle - Frontend: green/red Pipeline ON/OFF button on the pipeline page - Beta defaults to pipelineEnabled: false - Base values.yaml: blanked external URLs (Ollama, Polygon, Alpaca) so stages only connect to what they explicitly configure	2026-04-21 00:21:53 +00:00
Celes Renata	7071bba92d	fix: increase stale threshold to 4h to prevent duplicate enqueuing ... The 30-minute threshold was shorter than the queue drain time, causing the recovery sweep to re-enqueue docs that were already queued but not yet processed. Bumped to 4 hours with matching marker TTL.	2026-04-20 18:05:30 +00:00
Celes Renata	46c24aefab	fix: prevent duplicate queue entries with Redis SET markers ... Recovery sweeps and the retry endpoint now check a per-document Redis key (SET NX, 1h TTL) before pushing to the queue. If the marker exists, the doc is already enqueued and gets skipped. This prevents the scheduler from re-enqueuing the same parsed docs every 5 minutes.	2026-04-20 17:24:53 +00:00
Celes Renata	288c5333b5	fix: use queue_key() for stage-prefixed Redis queue names in pipeline endpoints ... The pipeline health, SSE stream, and retry endpoints were hardcoding 'stonks:queue:{name}' but services use DEPLOY_STAGE prefix ('stonks:paper:queue:{name}'). Now uses queue_key() from redis_keys.py.	2026-04-20 13:16:11 +00:00
Celes Renata	de35279269	feat: retry failed extractions button on pipeline page ... - POST /api/ops/pipeline/retry-failed endpoint resets extraction_failed docs to parsed, deletes failed intelligence rows, and re-enqueues them (batch of 200) - Scheduler now auto-retries extraction_failed docs every ~10 minutes (100 per cycle, 60-min cooldown per doc) - Pipeline page shows 'Retry Failed (N)' button when extraction_failed count > 0, with pending/success/error states	2026-04-20 08:09:29 +00:00
Celes Renata	950ff03f7e	fix: join through document_intelligence in patterns endpoint ... The inline catalyst_type query in GET /api/patterns/{ticker} referenced dir.document_id which does not exist on document_impact_records. The table links to documents via intelligence_id -> document_intelligence -> document_id. Added the missing JOIN to match the pattern used in _SELF_PATTERN_QUERY.	2026-04-20 07:12:13 +00:00
Celes Renata	5acb2fb43e	fix: resolve 6 integration test failures ... 1. patterns endpoint: fix query referencing non-existent column di.catalyst_type → dir.catalyst_type (column is on document_impact_records) 2. lockouts seed: use relative timestamps (now + 7d) so active lockout is always in the future regardless of when tests run 3. create_agent: make slug optional with auto-generation from name 4. create_source: json.dumps(config) + ::jsonb cast for asyncpg JSONB compat 5. approval_expiry: return count as int (len(expired)) not the list itself 6. metrics_consistency: fix test assertion to match API contract (total >= active + reserve, not total == active + reserve + unrealized)	2026-04-20 04:30:13 +00:00
Celes Renata	8f67d326c9	feat: derive POSTGRES_DB and Redis prefix from DEPLOY_STAGE for pipeline isolation	2026-04-20 01:33:14 +00:00
Celes Renata	d64ce82649	fix: scheduler timezone-aware datetime subtraction in is_source_due	2026-04-20 00:47:26 +00:00