Celes Renata
a72f336ad1
Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.
7.1 KiB
Bugfix Requirements Document
Introduction
Multiple operational bugs discovered in the stonks-beta namespace prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput. The core issue is that the outcome evaluation → metrics computation → quality gate pipeline is completely disconnected from the production scheduler, making the platform unable to self-calibrate or validate predictions. Additionally, Polygon API rate limiting causes ~40% request failures per cycle, a broken config query prevents the v3 engine from being toggled, several periodic snapshot tasks are missing from the scheduler, the lake-publisher deployment is idle/redundant, and order rejection reasons are lost.
Bug Analysis
Current Behavior (Defect)
1.1 WHEN the scheduler enqueues ingestion jobs for all 50 tickers' news_api and market_api sources simultaneously THEN the system exhausts the Polygon free-tier rate limit (5 req/min) resulting in ~40% of sources receiving HTTP 429 Too Many Requests every cycle
1.2 WHEN the aggregation worker reads the v3_engine_enabled flag via _V3_ENGINE_FLAG_QUERY THEN the system queries non-existent columns key and value on the risk_configs table (actual schema: name varchar, config JSONB) causing a PostgreSQL error every aggregation cycle
1.3 WHEN a scheduler cycle completes THEN the system never calls evaluate_matured_predictions() because it is not wired into the scheduler's main loop — only imported in backtest_replay.py
1.4 WHEN a scheduler cycle completes THEN the system never calls compute_and_store_metric_snapshots() because it is not wired into the scheduler's main loop — only called from backtest replay
1.5 WHEN the model quality gate evaluates trading eligibility THEN the system always fails with "no model metric snapshot available — defaulting to paper-only" because model_metric_snapshots table is permanently empty (consequence of bug 1.4)
1.6 WHEN the trading engine runs daily THEN the system never captures portfolio state snapshots to the portfolio_snapshots table because no periodic scheduler task invokes this capture
1.7 WHEN the trading engine runs daily THEN the system never captures risk state snapshots to the daily_risk_snapshots table because no periodic scheduler task invokes this capture
1.8 WHEN a prediction snapshot is created while Polygon rate-limiting has prevented the market data fetch THEN the system stores NULL in price_at_prediction (affecting 21% of snapshots), degrading downstream outcome evaluation accuracy
1.9 WHEN the standalone lake-publisher deployment polls stonks:beta:queue:lake_publish THEN the queue is always empty (0 items) because all lake publishing happens inline in broker-adapter and recommendation services — the deployment consumes zero work and wastes resources
1.10 WHEN Alpaca returns HTTP 401 for an order submission THEN the system sets order status to "rejected" but leaves the rejection_reason column NULL, capturing the error message only in the decision_trace JSONB field
Expected Behavior (Correct)
2.1 WHEN the scheduler enqueues ingestion jobs for Polygon-backed sources (news_api, market_api) THEN the system SHALL pace/stagger requests across the polling interval to stay within the Polygon rate limit, achieving near-zero 429 responses per cycle
2.2 WHEN the aggregation worker reads the v3_engine_enabled flag THEN the system SHALL query SELECT config FROM risk_configs WHERE name = 'v3_engine_enabled' and parse the JSONB value to determine the boolean toggle state
2.3 WHEN a scheduler cycle completes and sufficient time has elapsed since the last evaluation THEN the system SHALL call evaluate_matured_predictions() to evaluate prediction snapshots whose horizon has elapsed, populating the prediction_outcomes table
2.4 WHEN a scheduler cycle completes and sufficient time has elapsed since the last computation THEN the system SHALL call compute_and_store_metric_snapshots() to compute aggregate model metrics across all lookback/horizon combinations, populating model_metric_snapshots
2.5 WHEN the model quality gate evaluates trading eligibility THEN the system SHALL have recent metric snapshots available and evaluate thresholds against actual model performance data
2.6 WHEN market hours close (or on a daily schedule) THEN the system SHALL capture and persist the current portfolio state to portfolio_snapshots including value, returns, positions, and risk metrics
2.7 WHEN market hours close (or on a daily schedule) THEN the system SHALL capture and persist the current risk state to daily_risk_snapshots including portfolio value, daily P&L, trade count, and sector positions
2.8 WHEN a prediction snapshot is created and market price is unavailable due to rate limiting THEN the system SHALL retry the price fetch or defer the snapshot until price data is available, reducing NULL price_at_prediction occurrences to near zero
2.9 WHEN the lake-publisher deployment architecture is reviewed THEN the system SHALL either route lake publish jobs through the Redis queue to the standalone deployment, or remove the redundant deployment — eliminating the idle pod
2.10 WHEN Alpaca returns an HTTP error (401, 403, or any rejection) for an order submission THEN the system SHALL populate the rejection_reason column with the HTTP error message/status in addition to recording it in decision_trace
Unchanged Behavior (Regression Prevention)
3.1 WHEN sources with valid rate-limit headroom are enqueued THEN the system SHALL CONTINUE TO enqueue and process them without artificial delay
3.2 WHEN risk_configs is queried for other configuration keys (e.g., model_quality_gate_config, macro_enabled) THEN the system SHALL CONTINUE TO read them correctly using the existing name/config column pattern
3.3 WHEN the backtest replay module calls evaluate_matured_predictions() and compute_and_store_metric_snapshots() THEN the system SHALL CONTINUE TO execute them as part of backtest validation
3.4 WHEN prediction snapshots are created with available market prices THEN the system SHALL CONTINUE TO store the correct price_at_prediction value immediately
3.5 WHEN the existing inline lake publishing in broker-adapter and recommendation services writes facts THEN the system SHALL CONTINUE TO produce correct Parquet partitions in MinIO
3.6 WHEN orders succeed (HTTP 200 from Alpaca) THEN the system SHALL CONTINUE TO process them normally without modifying the rejection_reason column
3.7 WHEN the scheduler runs ingestion, extraction, aggregation, recommendation, and trading tasks THEN the system SHALL CONTINUE TO execute them on the existing cadence without disruption
3.8 WHEN the trading engine makes decisions and submits orders THEN the system SHALL CONTINUE TO record full decision context in decision_trace JSONB as before
3.9 WHEN the model quality gate passes (once metrics are populated) THEN the system SHALL CONTINUE TO allow promotion to live trading mode per existing threshold logic
3.10 WHEN the reporting collector fetches portfolio_snapshots and daily_risk_snapshots for report generation THEN the system SHALL CONTINUE TO query and render them using the existing schema