stonks-oracle/.kiro/specs/ops-pipeline-fixes/tasks.md

Implementation Plan: ops-pipeline-fixes

Overview

Fix 10 operational bugs preventing the validation/calibration feedback loop from functioning and degrading ingestion throughput. Changes span scheduler (rate limiting + periodic tasks), aggregation worker (config query), broker service (rejection reason), prediction snapshot (price fallback), and Helm chart (dead pod removal).

Tasks

• 1. Fix Polygon global rate limit — In services/scheduler/app.py, replace POLYGON_GLOBAL_RATE_LIMIT: int = 45 with POLYGON_GLOBAL_RATE_LIMIT: int = int(os.getenv("POLYGON_GLOBAL_RATE_LIMIT", "5")) to make it env-configurable and default to the free-tier limit

  • Validates: Bugfix 2.1; Regression 3.1, 3.7

• 2. Fix v3_engine_enabled query — In services/aggregation/worker.py, replace _V3_ENGINE_FLAG_QUERY from SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled' to SELECT config->>'v3_engine_enabled' AS enabled FROM risk_configs WHERE name = 'default' AND active = TRUE LIMIT 1, and rewrite _read_v3_flag() to parse the returned string (checking for "true"/"1"/"yes"), returning False for NULL/missing/error

  • Validates: Bugfix 2.2; Regression 3.2

• 3. Add validation cycle constant and counter — In services/scheduler/app.py, add VALIDATION_CYCLE_INTERVAL = int(os.getenv("VALIDATION_CYCLE_INTERVAL", "240")) constant and validation_counter = 0 initialization in main()

  • Validates: Bugfix 2.3, 2.4

• 4. Implement run_validation_cycle function — In services/scheduler/app.py, implement run_validation_cycle(pool) that calls evaluate_matured_predictions(pool) followed by compute_and_store_metric_snapshots(pool), with try/except logging for each and skipping metrics if outcomes fail

  • Validates: Bugfix 2.3, 2.4, 2.5; Regression 3.3

• 5. Wire validation cycle into main loop — In services/scheduler/app.py main loop, add the counter increment and conditional call to run_validation_cycle(pool) after the existing report_schedule_counter block

  • Validates: Bugfix 2.3, 2.4, 2.5

• 6. Add snapshot cycle constant and counter — In services/scheduler/app.py, add SNAPSHOT_CYCLE_INTERVAL = int(os.getenv("SNAPSHOT_CYCLE_INTERVAL", "240")) constant and snapshot_counter = 0 initialization in main()

  • Validates: Bugfix 2.6, 2.7

• 7. Implement maybe_capture_daily_snapshots function — In services/scheduler/app.py, implement maybe_capture_daily_snapshots(pool) that checks time (after 16:30 ET), checks idempotency (no existing row for today), queries positions table for portfolio value/unrealized PnL, and inserts into portfolio_snapshots and daily_risk_snapshots

  • Validates: Bugfix 2.6, 2.7; Regression 3.10

• 8. Wire snapshot cycle into main loop — In services/scheduler/app.py main loop, add the counter increment and conditional call to maybe_capture_daily_snapshots(pool) after the validation counter block

  • Validates: Bugfix 2.6, 2.7

• 9. Add prediction price fallback — In services/validation/prediction_snapshot.py, after the primary market_snapshots price lookup returns NULL for price_at_prediction, add a fallback query to positions table: SELECT current_price FROM positions WHERE ticker = $1 AND current_price IS NOT NULL LIMIT 1

  • Validates: Bugfix 2.8; Regression 3.4

• 10. Scale down lake-publisher — In infra/helm/stonks-oracle/values.yaml, change the lake-publisher replicas from 1 to 0

  • Validates: Bugfix 2.9; Regression 3.5

• 11. Extend _INSERT_ORDER SQL — In services/adapters/broker_service.py, extend _INSERT_ORDER SQL to include rejection_reason and rejected_at as parameters $18 and $19, with COALESCE in the ON CONFLICT UPDATE clause to preserve existing values

  • Validates: Bugfix 2.10; Regression 3.6, 3.8

• 12. Update persist_order parameters — In services/adapters/broker_service.py, update persist_order() to compute rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None and rejected_at = now if resp.status == OrderStatus.REJECTED else None, passing them as the final two parameters in the execute call

  • Validates: Bugfix 2.10; Regression 3.6, 3.8

• 13. Lint and test — Run .venv/bin/ruff check services/ and .venv/bin/python -m pytest tests/ -x --tb=short -q to verify no regressions

  • Validates: Regression 3.1–3.10

Task Dependency Graph

{
  "waves": [
    {"tasks": [1, 2, 9, 10]},
    {"tasks": [3, 6, 11]},
    {"tasks": [4, 7, 12]},
    {"tasks": [5, 8]},
    {"tasks": [13]}
  ]
}

Tasks 1, 2, 9, 10 are fully independent. Tasks 3/6/11 set up constants needed by 4/7/12. Tasks 5/8 wire into the main loop after their functions exist. Task 13 validates everything last.

Notes

• No database migrations required — all tables already exist with correct columns
• All scheduler changes use the existing counter-based periodic task pattern already established for cleanup, aggregation, and report tasks
• Lazy imports in run_validation_cycle avoid circular imports and keep scheduler startup fast
• The maybe_capture_daily_snapshots idempotency check prevents duplicate rows on scheduler restart