feat: Intelligence Pipeline v3 — full implementation
Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.
This commit is contained in:
Celes Renata
@@ -0,0 +1 @@
|
||||
{"specId": "f5d99301-94ef-4dc2-8ba4-ccefeee7ecba", "workflowType": "requirements-first", "specType": "bugfix"}
|
||||
@@ -0,0 +1,73 @@
|
||||
# Bugfix Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
Multiple operational bugs discovered in the stonks-beta namespace prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput. The core issue is that the outcome evaluation → metrics computation → quality gate pipeline is completely disconnected from the production scheduler, making the platform unable to self-calibrate or validate predictions. Additionally, Polygon API rate limiting causes ~40% request failures per cycle, a broken config query prevents the v3 engine from being toggled, several periodic snapshot tasks are missing from the scheduler, the lake-publisher deployment is idle/redundant, and order rejection reasons are lost.
|
||||
|
||||
## Bug Analysis
|
||||
|
||||
### Current Behavior (Defect)
|
||||
|
||||
1.1 WHEN the scheduler enqueues ingestion jobs for all 50 tickers' news_api and market_api sources simultaneously THEN the system exhausts the Polygon free-tier rate limit (5 req/min) resulting in ~40% of sources receiving HTTP 429 Too Many Requests every cycle
|
||||
|
||||
1.2 WHEN the aggregation worker reads the v3_engine_enabled flag via `_V3_ENGINE_FLAG_QUERY` THEN the system queries non-existent columns `key` and `value` on the `risk_configs` table (actual schema: `name` varchar, `config` JSONB) causing a PostgreSQL error every aggregation cycle
|
||||
|
||||
1.3 WHEN a scheduler cycle completes THEN the system never calls `evaluate_matured_predictions()` because it is not wired into the scheduler's main loop — only imported in `backtest_replay.py`
|
||||
|
||||
1.4 WHEN a scheduler cycle completes THEN the system never calls `compute_and_store_metric_snapshots()` because it is not wired into the scheduler's main loop — only called from backtest replay
|
||||
|
||||
1.5 WHEN the model quality gate evaluates trading eligibility THEN the system always fails with "no model metric snapshot available — defaulting to paper-only" because `model_metric_snapshots` table is permanently empty (consequence of bug 1.4)
|
||||
|
||||
1.6 WHEN the trading engine runs daily THEN the system never captures portfolio state snapshots to the `portfolio_snapshots` table because no periodic scheduler task invokes this capture
|
||||
|
||||
1.7 WHEN the trading engine runs daily THEN the system never captures risk state snapshots to the `daily_risk_snapshots` table because no periodic scheduler task invokes this capture
|
||||
|
||||
1.8 WHEN a prediction snapshot is created while Polygon rate-limiting has prevented the market data fetch THEN the system stores NULL in `price_at_prediction` (affecting 21% of snapshots), degrading downstream outcome evaluation accuracy
|
||||
|
||||
1.9 WHEN the standalone `lake-publisher` deployment polls `stonks:beta:queue:lake_publish` THEN the queue is always empty (0 items) because all lake publishing happens inline in broker-adapter and recommendation services — the deployment consumes zero work and wastes resources
|
||||
|
||||
1.10 WHEN Alpaca returns HTTP 401 for an order submission THEN the system sets order status to "rejected" but leaves the `rejection_reason` column NULL, capturing the error message only in the `decision_trace` JSONB field
|
||||
|
||||
### Expected Behavior (Correct)
|
||||
|
||||
2.1 WHEN the scheduler enqueues ingestion jobs for Polygon-backed sources (news_api, market_api) THEN the system SHALL pace/stagger requests across the polling interval to stay within the Polygon rate limit, achieving near-zero 429 responses per cycle
|
||||
|
||||
2.2 WHEN the aggregation worker reads the v3_engine_enabled flag THEN the system SHALL query `SELECT config FROM risk_configs WHERE name = 'v3_engine_enabled'` and parse the JSONB value to determine the boolean toggle state
|
||||
|
||||
2.3 WHEN a scheduler cycle completes and sufficient time has elapsed since the last evaluation THEN the system SHALL call `evaluate_matured_predictions()` to evaluate prediction snapshots whose horizon has elapsed, populating the `prediction_outcomes` table
|
||||
|
||||
2.4 WHEN a scheduler cycle completes and sufficient time has elapsed since the last computation THEN the system SHALL call `compute_and_store_metric_snapshots()` to compute aggregate model metrics across all lookback/horizon combinations, populating `model_metric_snapshots`
|
||||
|
||||
2.5 WHEN the model quality gate evaluates trading eligibility THEN the system SHALL have recent metric snapshots available and evaluate thresholds against actual model performance data
|
||||
|
||||
2.6 WHEN market hours close (or on a daily schedule) THEN the system SHALL capture and persist the current portfolio state to `portfolio_snapshots` including value, returns, positions, and risk metrics
|
||||
|
||||
2.7 WHEN market hours close (or on a daily schedule) THEN the system SHALL capture and persist the current risk state to `daily_risk_snapshots` including portfolio value, daily P&L, trade count, and sector positions
|
||||
|
||||
2.8 WHEN a prediction snapshot is created and market price is unavailable due to rate limiting THEN the system SHALL retry the price fetch or defer the snapshot until price data is available, reducing NULL `price_at_prediction` occurrences to near zero
|
||||
|
||||
2.9 WHEN the lake-publisher deployment architecture is reviewed THEN the system SHALL either route lake publish jobs through the Redis queue to the standalone deployment, or remove the redundant deployment — eliminating the idle pod
|
||||
|
||||
2.10 WHEN Alpaca returns an HTTP error (401, 403, or any rejection) for an order submission THEN the system SHALL populate the `rejection_reason` column with the HTTP error message/status in addition to recording it in `decision_trace`
|
||||
|
||||
### Unchanged Behavior (Regression Prevention)
|
||||
|
||||
3.1 WHEN sources with valid rate-limit headroom are enqueued THEN the system SHALL CONTINUE TO enqueue and process them without artificial delay
|
||||
|
||||
3.2 WHEN risk_configs is queried for other configuration keys (e.g., `model_quality_gate_config`, `macro_enabled`) THEN the system SHALL CONTINUE TO read them correctly using the existing `name`/`config` column pattern
|
||||
|
||||
3.3 WHEN the backtest replay module calls `evaluate_matured_predictions()` and `compute_and_store_metric_snapshots()` THEN the system SHALL CONTINUE TO execute them as part of backtest validation
|
||||
|
||||
3.4 WHEN prediction snapshots are created with available market prices THEN the system SHALL CONTINUE TO store the correct `price_at_prediction` value immediately
|
||||
|
||||
3.5 WHEN the existing inline lake publishing in broker-adapter and recommendation services writes facts THEN the system SHALL CONTINUE TO produce correct Parquet partitions in MinIO
|
||||
|
||||
3.6 WHEN orders succeed (HTTP 200 from Alpaca) THEN the system SHALL CONTINUE TO process them normally without modifying the `rejection_reason` column
|
||||
|
||||
3.7 WHEN the scheduler runs ingestion, extraction, aggregation, recommendation, and trading tasks THEN the system SHALL CONTINUE TO execute them on the existing cadence without disruption
|
||||
|
||||
3.8 WHEN the trading engine makes decisions and submits orders THEN the system SHALL CONTINUE TO record full decision context in `decision_trace` JSONB as before
|
||||
|
||||
3.9 WHEN the model quality gate passes (once metrics are populated) THEN the system SHALL CONTINUE TO allow promotion to live trading mode per existing threshold logic
|
||||
|
||||
3.10 WHEN the reporting collector fetches portfolio_snapshots and daily_risk_snapshots for report generation THEN the system SHALL CONTINUE TO query and render them using the existing schema
|
||||
@@ -0,0 +1,396 @@
|
||||
# Technical Design: ops-pipeline-fixes
|
||||
|
||||
## Overview
|
||||
|
||||
This design addresses 10 operational bugs that prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput in the `stonks-beta` namespace. The fixes span the scheduler (rate limiting + periodic tasks), aggregation worker (config query), broker service (rejection reason), prediction snapshot (price fallback), and Helm chart (dead pod removal). All changes are localized with graceful fallbacks and no schema migrations required.
|
||||
|
||||
## Bug Details
|
||||
|
||||
Multiple operational bugs in the `stonks-beta` namespace prevent the validation/calibration feedback loop from functioning and degrade ingestion throughput. The core pipeline (ingestion → extraction → aggregation → recommendation → trading) flows end-to-end, but:
|
||||
- The outcome evaluation → metrics computation → quality gate feedback loop is completely disconnected
|
||||
- Polygon API rate limiting causes ~40% ingestion failures per cycle
|
||||
- A broken config query prevents the v3 engine toggle from working
|
||||
- Portfolio/risk snapshots are never captured
|
||||
- Order rejection reasons are lost
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Scheduler (services/scheduler/app.py)"
|
||||
A[schedule_cycle] -->|paced enqueue| B[Ingestion Queue]
|
||||
C[validation_cycle] -->|hourly| D[evaluate_matured_predictions]
|
||||
C -->|after outcomes| E[compute_and_store_metric_snapshots]
|
||||
F[snapshot_cycle] -->|daily 16:30 ET| G[capture_portfolio_snapshot]
|
||||
F -->|daily 16:30 ET| H[capture_risk_snapshot]
|
||||
end
|
||||
|
||||
subgraph "Aggregation (services/aggregation/worker.py)"
|
||||
I[_read_v3_flag] -->|fixed query| J[risk_configs.config JSONB]
|
||||
end
|
||||
|
||||
subgraph "Broker (services/adapters/broker_service.py)"
|
||||
K[persist_order] -->|rejected status| L[orders.rejection_reason]
|
||||
end
|
||||
|
||||
D --> O[prediction_outcomes]
|
||||
E --> P[model_metric_snapshots]
|
||||
P --> Q[Quality Gate]
|
||||
```
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
2.1 The scheduler SHALL pace Polygon API requests within the free-tier limit (~5 req/min), achieving near-zero 429 responses per cycle.
|
||||
|
||||
2.2 The aggregation worker SHALL read `v3_engine_enabled` from the `risk_configs` JSONB `config` column (not non-existent `key`/`value` columns).
|
||||
|
||||
2.3 The scheduler SHALL call `evaluate_matured_predictions()` hourly to populate `prediction_outcomes`.
|
||||
|
||||
2.4 The scheduler SHALL call `compute_and_store_metric_snapshots()` after outcome evaluation to populate `model_metric_snapshots`.
|
||||
|
||||
2.5 The quality gate SHALL have recent metric data available once the validation cycle runs.
|
||||
|
||||
2.6 The scheduler SHALL capture daily portfolio snapshots to `portfolio_snapshots` after market close.
|
||||
|
||||
2.7 The scheduler SHALL capture daily risk snapshots to `daily_risk_snapshots` after market close.
|
||||
|
||||
2.8 Prediction snapshots SHALL fall back to positions table prices when market_snapshots data is unavailable.
|
||||
|
||||
2.9 The lake-publisher deployment SHALL be scaled to 0 (idle pod, wasted resources).
|
||||
|
||||
2.10 The broker service SHALL populate `rejection_reason` on orders when broker or risk engine rejects.
|
||||
|
||||
## Hypothesized Root Cause
|
||||
|
||||
### Bug 1.1 — Polygon Rate Limiting
|
||||
`POLYGON_GLOBAL_RATE_LIMIT = 45` in `services/scheduler/app.py` is set for a paid Polygon plan but the deployed instance uses the free tier (5 req/min). All 50+ sources are attempted per cycle, exhausting the limit instantly.
|
||||
|
||||
### Bug 1.2 — v3_engine_enabled Config Read
|
||||
`_V3_ENGINE_FLAG_QUERY` in `services/aggregation/worker.py` reads `SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'`. The actual table has columns `name` (varchar) and `config` (JSONB) — no `key` or `value` column exists.
|
||||
|
||||
### Bug 1.3 & 1.4 — Outcome Evaluator & Metrics Never Scheduled
|
||||
`evaluate_matured_predictions()` and `compute_and_store_metric_snapshots()` exist in `services/validation/` but are only imported in `services/trading/backtest_replay.py`. The scheduler main loop in `services/scheduler/app.py` has no call to either function.
|
||||
|
||||
### Bug 1.5 — Quality Gate Permanently Failing
|
||||
`services/trading/model_quality_gate.py` queries `model_metric_snapshots` which is always empty (consequence of 1.4). Returns "no model metric snapshot available — defaulting to paper-only" every time.
|
||||
|
||||
### Bug 1.6 & 1.7 — Portfolio/Risk Snapshots
|
||||
The trading engine has `_persist_daily_snapshot()` but it only executes when the engine's main loop is actively processing trades. The trading-engine pod shows only health checks — its main loop isn't cycling because there are no active trade triggers flowing through it. No fallback capture exists in the scheduler.
|
||||
|
||||
### Bug 1.8 — Market Price Gaps
|
||||
`services/validation/prediction_snapshot.py` queries `market_snapshots` for price at prediction time. When Polygon rate limiting prevents market data fetches, no snapshot exists and `price_at_prediction` is NULL. 21% of snapshots affected.
|
||||
|
||||
### Bug 1.9 — Lake Publisher Idle
|
||||
The standalone `lake-publisher` deployment polls `stonks:beta:queue:lake_publish` but all services (broker-adapter, recommendation) import `services.lake_publisher.worker` directly and publish inline — never pushing to the Redis queue.
|
||||
|
||||
### Bug 1.10 — Order rejection_reason NULL
|
||||
`_INSERT_ORDER` SQL in `services/adapters/broker_service.py` doesn't include `rejection_reason` or `rejected_at` columns. The error is stored in `decision_trace` JSONB but the dedicated column stays NULL. The reconciliation path (`_reconcile_open_orders`) does set these columns, but initial persist does not.
|
||||
|
||||
## Fix Implementation
|
||||
|
||||
### Fix 1: Polygon Rate Limit Constant (Bug 1.1)
|
||||
|
||||
**File:** `services/scheduler/app.py`
|
||||
|
||||
Replace the hardcoded constant with an env-configurable value defaulting to 5:
|
||||
|
||||
```python
|
||||
# Before:
|
||||
POLYGON_GLOBAL_RATE_LIMIT: int = 45
|
||||
|
||||
# After:
|
||||
POLYGON_GLOBAL_RATE_LIMIT: int = int(os.getenv("POLYGON_GLOBAL_RATE_LIMIT", "5"))
|
||||
```
|
||||
|
||||
The existing `check_rate_limit()` function already implements per-minute windowed counting and skips sources once the limit is hit. By reducing the constant to match the free-tier limit, the system will naturally pace — enqueuing ~5 Polygon sources per minute across scheduler ticks (15s interval = 4 ticks/min). Skipped sources are retried next cycle.
|
||||
|
||||
**Validates:** Bugfix 2.1; Regression 3.1, 3.7
|
||||
|
||||
---
|
||||
|
||||
### Fix 2: v3_engine_enabled Config Query (Bug 1.2)
|
||||
|
||||
**File:** `services/aggregation/worker.py`
|
||||
|
||||
Replace the broken query and function:
|
||||
|
||||
```python
|
||||
# Before:
|
||||
_V3_ENGINE_FLAG_QUERY = """
|
||||
SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'
|
||||
"""
|
||||
|
||||
# After:
|
||||
_V3_ENGINE_FLAG_QUERY = """
|
||||
SELECT config->>'v3_engine_enabled' AS enabled
|
||||
FROM risk_configs
|
||||
WHERE name = 'default' AND active = TRUE
|
||||
LIMIT 1
|
||||
"""
|
||||
|
||||
async def _read_v3_flag(pool: asyncpg.Pool) -> bool:
|
||||
"""Read v3_engine_enabled from risk_configs JSONB. Default False on error."""
|
||||
try:
|
||||
row = await pool.fetchrow(_V3_ENGINE_FLAG_QUERY)
|
||||
if row and row["enabled"]:
|
||||
return row["enabled"].lower() in ("true", "1", "yes")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning("Failed to read v3_engine_enabled flag: %s", e)
|
||||
return False
|
||||
```
|
||||
|
||||
Reads from the `default` active risk_config's JSONB `config` field. Falls back to False (unchanged fail-safe).
|
||||
|
||||
**Validates:** Bugfix 2.2; Regression 3.2
|
||||
|
||||
---
|
||||
|
||||
### Fix 3: Validation Cycle in Scheduler (Bugs 1.3, 1.4, 1.5)
|
||||
|
||||
**File:** `services/scheduler/app.py`
|
||||
|
||||
Add a new periodic task (every ~240 ticks = ~60 minutes):
|
||||
|
||||
```python
|
||||
# New constant:
|
||||
VALIDATION_CYCLE_INTERVAL = int(os.getenv("VALIDATION_CYCLE_INTERVAL", "240"))
|
||||
|
||||
# New counter in main():
|
||||
validation_counter = 0
|
||||
|
||||
# In main loop after existing periodic tasks:
|
||||
validation_counter += 1
|
||||
if validation_counter >= VALIDATION_CYCLE_INTERVAL:
|
||||
validation_counter = 0
|
||||
await run_validation_cycle(pool)
|
||||
```
|
||||
|
||||
New function:
|
||||
|
||||
```python
|
||||
async def run_validation_cycle(pool: asyncpg.Pool) -> None:
|
||||
"""Run outcome evaluation and metric computation (hourly).
|
||||
|
||||
Requirements: 2.3, 2.4, 2.5
|
||||
"""
|
||||
from services.validation.outcome_evaluator import evaluate_matured_predictions
|
||||
from services.validation.metrics import compute_and_store_metric_snapshots
|
||||
|
||||
try:
|
||||
outcomes = await evaluate_matured_predictions(pool)
|
||||
logger.info("Validation: evaluated %d prediction outcomes", outcomes)
|
||||
except Exception:
|
||||
logger.exception("Validation: outcome evaluation failed")
|
||||
return # Skip metrics if outcomes failed
|
||||
|
||||
try:
|
||||
snapshots = await compute_and_store_metric_snapshots(pool)
|
||||
logger.info("Validation: computed %d metric snapshots", len(snapshots))
|
||||
except Exception:
|
||||
logger.exception("Validation: metric computation failed")
|
||||
```
|
||||
|
||||
**Validates:** Bugfix 2.3, 2.4, 2.5; Regression 3.3
|
||||
|
||||
---
|
||||
|
||||
### Fix 4: Daily Portfolio & Risk Snapshots (Bugs 1.6, 1.7)
|
||||
|
||||
**File:** `services/scheduler/app.py`
|
||||
|
||||
Add a daily snapshot task that runs every ~60 minutes but only captures once per day after 16:30 ET:
|
||||
|
||||
```python
|
||||
SNAPSHOT_CYCLE_INTERVAL = int(os.getenv("SNAPSHOT_CYCLE_INTERVAL", "240"))
|
||||
snapshot_counter = 0
|
||||
|
||||
# In main loop:
|
||||
snapshot_counter += 1
|
||||
if snapshot_counter >= SNAPSHOT_CYCLE_INTERVAL:
|
||||
snapshot_counter = 0
|
||||
await maybe_capture_daily_snapshots(pool)
|
||||
```
|
||||
|
||||
New function:
|
||||
|
||||
```python
|
||||
async def maybe_capture_daily_snapshots(pool: asyncpg.Pool) -> None:
|
||||
"""Capture portfolio and risk snapshots once daily after market close.
|
||||
|
||||
Requirements: 2.6, 2.7
|
||||
"""
|
||||
et_now = datetime.now(ZoneInfo("America/New_York"))
|
||||
|
||||
# Only after 4:30 PM ET
|
||||
if et_now.hour < 16 or (et_now.hour == 16 and et_now.minute < 30):
|
||||
return
|
||||
|
||||
today = et_now.date()
|
||||
|
||||
# Already captured today?
|
||||
existing = await pool.fetchval(
|
||||
"SELECT 1 FROM portfolio_snapshots WHERE snapshot_date = $1 LIMIT 1",
|
||||
today,
|
||||
)
|
||||
if existing:
|
||||
return
|
||||
|
||||
# Portfolio snapshot from positions + account data
|
||||
try:
|
||||
positions = await pool.fetch("SELECT * FROM positions WHERE quantity > 0")
|
||||
portfolio_value = sum(
|
||||
float(r["current_price"] or 0) * float(r["quantity"])
|
||||
for r in positions
|
||||
)
|
||||
unrealized_pnl = sum(float(r["unrealized_pnl"] or 0) for r in positions)
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO portfolio_snapshots
|
||||
(snapshot_date, portfolio_value, unrealized_pnl, positions)
|
||||
VALUES ($1, $2, $3, $4::jsonb)""",
|
||||
today, portfolio_value, unrealized_pnl,
|
||||
json.dumps([dict(r) for r in positions], default=str),
|
||||
)
|
||||
logger.info("Captured portfolio snapshot: value=%.2f", portfolio_value)
|
||||
except Exception:
|
||||
logger.exception("Failed to capture portfolio snapshot")
|
||||
|
||||
# Risk snapshot from daily activity
|
||||
try:
|
||||
daily_orders = await pool.fetchval(
|
||||
"SELECT count(*) FROM orders WHERE created_at::date = $1", today
|
||||
)
|
||||
daily_pnl = sum(float(r["unrealized_pnl"] or 0) for r in positions) if positions else 0.0
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO daily_risk_snapshots
|
||||
(account_id, snapshot_date, portfolio_value, daily_pnl, daily_trade_count)
|
||||
VALUES ((SELECT id FROM broker_accounts LIMIT 1), $1, $2, $3, $4)
|
||||
ON CONFLICT DO NOTHING""",
|
||||
today, portfolio_value, daily_pnl, daily_orders or 0,
|
||||
)
|
||||
logger.info("Captured risk snapshot: pnl=%.2f trades=%d", daily_pnl, daily_orders or 0)
|
||||
except Exception:
|
||||
logger.exception("Failed to capture risk snapshot")
|
||||
```
|
||||
|
||||
**Validates:** Bugfix 2.6, 2.7; Regression 3.10
|
||||
|
||||
---
|
||||
|
||||
### Fix 5: Prediction Price Fallback (Bug 1.8)
|
||||
|
||||
**File:** `services/validation/prediction_snapshot.py`
|
||||
|
||||
After the primary `market_snapshots` price lookup returns NULL, add a fallback:
|
||||
|
||||
```python
|
||||
# After market_snapshots lookup:
|
||||
if price_at_prediction is None:
|
||||
pos_row = await conn.fetchrow(
|
||||
"SELECT current_price FROM positions "
|
||||
"WHERE ticker = $1 AND current_price IS NOT NULL LIMIT 1",
|
||||
ticker,
|
||||
)
|
||||
if pos_row:
|
||||
price_at_prediction = float(pos_row["current_price"])
|
||||
```
|
||||
|
||||
Only covers tickers with open positions (currently 10). Acceptable tradeoff — most active tickers are the ones we hold.
|
||||
|
||||
**Validates:** Bugfix 2.8; Regression 3.4
|
||||
|
||||
---
|
||||
|
||||
### Fix 6: Lake Publisher Scale-Down (Bug 1.9)
|
||||
|
||||
**File:** `infra/helm/stonks-oracle/values.yaml`
|
||||
|
||||
```yaml
|
||||
# Change:
|
||||
replicas: 0
|
||||
```
|
||||
|
||||
Keeps the deployment definition intact for future use but schedules no pods.
|
||||
|
||||
**Validates:** Bugfix 2.9; Regression 3.5
|
||||
|
||||
---
|
||||
|
||||
### Fix 7: Order rejection_reason Population (Bug 1.10)
|
||||
|
||||
**File:** `services/adapters/broker_service.py`
|
||||
|
||||
Extend `_INSERT_ORDER` to include `rejection_reason` and `rejected_at`:
|
||||
|
||||
```python
|
||||
_INSERT_ORDER = """
|
||||
INSERT INTO orders (
|
||||
id, recommendation_id, broker_account_id, ticker, side, order_type,
|
||||
quantity, limit_price, stop_price, status, idempotency_key,
|
||||
broker_order_id, decision_trace, submitted_at, filled_at,
|
||||
fill_price, fill_quantity, rejection_reason, rejected_at
|
||||
) VALUES (
|
||||
$1::uuid, $2, $3::uuid, $4, $5, $6,
|
||||
$7, $8, $9, $10, $11,
|
||||
$12, $13::jsonb, $14, $15,
|
||||
$16, $17, $18, $19
|
||||
)
|
||||
ON CONFLICT (idempotency_key) DO UPDATE SET
|
||||
status = EXCLUDED.status,
|
||||
broker_order_id = EXCLUDED.broker_order_id,
|
||||
filled_at = EXCLUDED.filled_at,
|
||||
fill_price = EXCLUDED.fill_price,
|
||||
fill_quantity = EXCLUDED.fill_quantity,
|
||||
rejection_reason = COALESCE(EXCLUDED.rejection_reason, orders.rejection_reason),
|
||||
rejected_at = COALESCE(EXCLUDED.rejected_at, orders.rejected_at),
|
||||
updated_at = NOW()
|
||||
"""
|
||||
```
|
||||
|
||||
Update `persist_order()` to pass the new parameters:
|
||||
|
||||
```python
|
||||
rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None
|
||||
rejected_at = now if resp.status == OrderStatus.REJECTED else None
|
||||
# Add as params $18, $19
|
||||
```
|
||||
|
||||
**Validates:** Bugfix 2.10; Regression 3.6, 3.8
|
||||
|
||||
---
|
||||
|
||||
## Correctness Properties
|
||||
|
||||
Property 1: Rate limit compliance — After fix, the rolling 1-minute window for Polygon requests SHALL NOT exceed the configured limit (default 5). Existing `check_rate_limit()` windowed counter enforces this; we only change the threshold constant.
|
||||
|
||||
Property 2: Validation cycle completeness — `prediction_outcomes` row count SHALL grow monotonically after the first validation cycle runs. Each run finds matured snapshots not yet evaluated and persists outcomes.
|
||||
|
||||
Property 3: Metric snapshot freshness — `model_metric_snapshots` SHALL contain rows with `generated_at` within the last 2 hours after 2+ validation cycles. The quality gate can then evaluate against real data.
|
||||
|
||||
Property 4: Config read correctness — `_read_v3_flag()` SHALL return True when `risk_configs.config->>'v3_engine_enabled'` is `'true'` and False for all other values including NULL or missing key.
|
||||
|
||||
Property 5: Snapshot idempotency — `portfolio_snapshots` SHALL contain at most 1 row per `snapshot_date`. The `maybe_capture_daily_snapshots` function checks for existing rows before insert.
|
||||
|
||||
Property 6: Rejection reason preservation — Every order with `status = 'rejected'` persisted via `persist_order()` SHALL have a non-NULL `rejection_reason` extracted from the error response.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests:** Update `test_scheduler.py` with a test verifying `run_validation_cycle` is called after the counter threshold. Test `_read_v3_flag` with mocked JSONB config returning various values.
|
||||
- **Integration tests:** Verify `persist_order` with rejected status populates `rejection_reason` column.
|
||||
- **Manual verification post-deploy:**
|
||||
- `kubectl logs deployment/scheduler -n stonks-beta --tail=100 | grep Validation` shows outcome counts
|
||||
- `SELECT count(*) FROM prediction_outcomes` starts growing within 1 hour
|
||||
- `SELECT count(*) FROM model_metric_snapshots` populates after outcomes exist
|
||||
- Scheduler logs show significantly fewer "Rate limit hit" warnings
|
||||
- Aggregation logs no longer show "column value does not exist" error
|
||||
- After market close: `SELECT * FROM portfolio_snapshots WHERE snapshot_date = CURRENT_DATE` returns 1 row
|
||||
|
||||
## Glossary
|
||||
|
||||
| Term | Definition |
|
||||
|------|-----------|
|
||||
| Validation cycle | Hourly scheduler task: evaluate_matured_predictions → compute_and_store_metric_snapshots |
|
||||
| Quality gate | Threshold check on model_metric_snapshots that determines if trading can be promoted from paper to live |
|
||||
| Prediction snapshot | Frozen state of a recommendation at generation time (prices, evidence, scores) |
|
||||
| Outcome evaluation | Matching a matured prediction snapshot against realized market returns |
|
||||
| Polygon free tier | API plan with ~5 requests/minute rate limit |
|
||||
@@ -0,0 +1,69 @@
|
||||
# Implementation Plan: ops-pipeline-fixes
|
||||
|
||||
## Overview
|
||||
|
||||
Fix 10 operational bugs preventing the validation/calibration feedback loop from functioning and degrading ingestion throughput. Changes span scheduler (rate limiting + periodic tasks), aggregation worker (config query), broker service (rejection reason), prediction snapshot (price fallback), and Helm chart (dead pod removal).
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] 1. Fix Polygon global rate limit — In `services/scheduler/app.py`, replace `POLYGON_GLOBAL_RATE_LIMIT: int = 45` with `POLYGON_GLOBAL_RATE_LIMIT: int = int(os.getenv("POLYGON_GLOBAL_RATE_LIMIT", "5"))` to make it env-configurable and default to the free-tier limit
|
||||
- **Validates: Bugfix 2.1; Regression 3.1, 3.7**
|
||||
|
||||
- [x] 2. Fix v3_engine_enabled query — In `services/aggregation/worker.py`, replace `_V3_ENGINE_FLAG_QUERY` from `SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'` to `SELECT config->>'v3_engine_enabled' AS enabled FROM risk_configs WHERE name = 'default' AND active = TRUE LIMIT 1`, and rewrite `_read_v3_flag()` to parse the returned string (checking for "true"/"1"/"yes"), returning False for NULL/missing/error
|
||||
- **Validates: Bugfix 2.2; Regression 3.2**
|
||||
|
||||
- [x] 3. Add validation cycle constant and counter — In `services/scheduler/app.py`, add `VALIDATION_CYCLE_INTERVAL = int(os.getenv("VALIDATION_CYCLE_INTERVAL", "240"))` constant and `validation_counter = 0` initialization in `main()`
|
||||
- **Validates: Bugfix 2.3, 2.4**
|
||||
|
||||
- [x] 4. Implement run_validation_cycle function — In `services/scheduler/app.py`, implement `run_validation_cycle(pool)` that calls `evaluate_matured_predictions(pool)` followed by `compute_and_store_metric_snapshots(pool)`, with try/except logging for each and skipping metrics if outcomes fail
|
||||
- **Validates: Bugfix 2.3, 2.4, 2.5; Regression 3.3**
|
||||
|
||||
- [x] 5. Wire validation cycle into main loop — In `services/scheduler/app.py` main loop, add the counter increment and conditional call to `run_validation_cycle(pool)` after the existing `report_schedule_counter` block
|
||||
- **Validates: Bugfix 2.3, 2.4, 2.5**
|
||||
|
||||
- [x] 6. Add snapshot cycle constant and counter — In `services/scheduler/app.py`, add `SNAPSHOT_CYCLE_INTERVAL = int(os.getenv("SNAPSHOT_CYCLE_INTERVAL", "240"))` constant and `snapshot_counter = 0` initialization in `main()`
|
||||
- **Validates: Bugfix 2.6, 2.7**
|
||||
|
||||
- [x] 7. Implement maybe_capture_daily_snapshots function — In `services/scheduler/app.py`, implement `maybe_capture_daily_snapshots(pool)` that checks time (after 16:30 ET), checks idempotency (no existing row for today), queries positions table for portfolio value/unrealized PnL, and inserts into `portfolio_snapshots` and `daily_risk_snapshots`
|
||||
- **Validates: Bugfix 2.6, 2.7; Regression 3.10**
|
||||
|
||||
- [x] 8. Wire snapshot cycle into main loop — In `services/scheduler/app.py` main loop, add the counter increment and conditional call to `maybe_capture_daily_snapshots(pool)` after the validation counter block
|
||||
- **Validates: Bugfix 2.6, 2.7**
|
||||
|
||||
- [x] 9. Add prediction price fallback — In `services/validation/prediction_snapshot.py`, after the primary market_snapshots price lookup returns NULL for `price_at_prediction`, add a fallback query to positions table: `SELECT current_price FROM positions WHERE ticker = $1 AND current_price IS NOT NULL LIMIT 1`
|
||||
- **Validates: Bugfix 2.8; Regression 3.4**
|
||||
|
||||
- [x] 10. Scale down lake-publisher — In `infra/helm/stonks-oracle/values.yaml`, change the lake-publisher `replicas` from `1` to `0`
|
||||
- **Validates: Bugfix 2.9; Regression 3.5**
|
||||
|
||||
- [x] 11. Extend _INSERT_ORDER SQL — In `services/adapters/broker_service.py`, extend `_INSERT_ORDER` SQL to include `rejection_reason` and `rejected_at` as parameters $18 and $19, with COALESCE in the ON CONFLICT UPDATE clause to preserve existing values
|
||||
- **Validates: Bugfix 2.10; Regression 3.6, 3.8**
|
||||
|
||||
- [x] 12. Update persist_order parameters — In `services/adapters/broker_service.py`, update `persist_order()` to compute `rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None` and `rejected_at = now if resp.status == OrderStatus.REJECTED else None`, passing them as the final two parameters in the execute call
|
||||
- **Validates: Bugfix 2.10; Regression 3.6, 3.8**
|
||||
|
||||
- [x] 13. Lint and test — Run `.venv/bin/ruff check services/` and `.venv/bin/python -m pytest tests/ -x --tb=short -q` to verify no regressions
|
||||
- **Validates: Regression 3.1–3.10**
|
||||
|
||||
## Task Dependency Graph
|
||||
|
||||
```json
|
||||
{
|
||||
"waves": [
|
||||
{"tasks": [1, 2, 9, 10]},
|
||||
{"tasks": [3, 6, 11]},
|
||||
{"tasks": [4, 7, 12]},
|
||||
{"tasks": [5, 8]},
|
||||
{"tasks": [13]}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Tasks 1, 2, 9, 10 are fully independent. Tasks 3/6/11 set up constants needed by 4/7/12. Tasks 5/8 wire into the main loop after their functions exist. Task 13 validates everything last.
|
||||
|
||||
## Notes
|
||||
|
||||
- No database migrations required — all tables already exist with correct columns
|
||||
- All scheduler changes use the existing counter-based periodic task pattern already established for cleanup, aggregation, and report tasks
|
||||
- Lazy imports in `run_validation_cycle` avoid circular imports and keep scheduler startup fast
|
||||
- The `maybe_capture_daily_snapshots` idempotency check prevents duplicate rows on scheduler restart
|
||||
Reference in New Issue
Block a user