Celes Renata
a72f336ad1
Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.
70 lines
5.2 KiB
Markdown
70 lines
5.2 KiB
Markdown
# Implementation Plan: ops-pipeline-fixes
|
||
|
||
## Overview
|
||
|
||
Fix 10 operational bugs preventing the validation/calibration feedback loop from functioning and degrading ingestion throughput. Changes span scheduler (rate limiting + periodic tasks), aggregation worker (config query), broker service (rejection reason), prediction snapshot (price fallback), and Helm chart (dead pod removal).
|
||
|
||
## Tasks
|
||
|
||
- [x] 1. Fix Polygon global rate limit — In `services/scheduler/app.py`, replace `POLYGON_GLOBAL_RATE_LIMIT: int = 45` with `POLYGON_GLOBAL_RATE_LIMIT: int = int(os.getenv("POLYGON_GLOBAL_RATE_LIMIT", "5"))` to make it env-configurable and default to the free-tier limit
|
||
- **Validates: Bugfix 2.1; Regression 3.1, 3.7**
|
||
|
||
- [x] 2. Fix v3_engine_enabled query — In `services/aggregation/worker.py`, replace `_V3_ENGINE_FLAG_QUERY` from `SELECT value FROM risk_configs WHERE key = 'v3_engine_enabled'` to `SELECT config->>'v3_engine_enabled' AS enabled FROM risk_configs WHERE name = 'default' AND active = TRUE LIMIT 1`, and rewrite `_read_v3_flag()` to parse the returned string (checking for "true"/"1"/"yes"), returning False for NULL/missing/error
|
||
- **Validates: Bugfix 2.2; Regression 3.2**
|
||
|
||
- [x] 3. Add validation cycle constant and counter — In `services/scheduler/app.py`, add `VALIDATION_CYCLE_INTERVAL = int(os.getenv("VALIDATION_CYCLE_INTERVAL", "240"))` constant and `validation_counter = 0` initialization in `main()`
|
||
- **Validates: Bugfix 2.3, 2.4**
|
||
|
||
- [x] 4. Implement run_validation_cycle function — In `services/scheduler/app.py`, implement `run_validation_cycle(pool)` that calls `evaluate_matured_predictions(pool)` followed by `compute_and_store_metric_snapshots(pool)`, with try/except logging for each and skipping metrics if outcomes fail
|
||
- **Validates: Bugfix 2.3, 2.4, 2.5; Regression 3.3**
|
||
|
||
- [x] 5. Wire validation cycle into main loop — In `services/scheduler/app.py` main loop, add the counter increment and conditional call to `run_validation_cycle(pool)` after the existing `report_schedule_counter` block
|
||
- **Validates: Bugfix 2.3, 2.4, 2.5**
|
||
|
||
- [x] 6. Add snapshot cycle constant and counter — In `services/scheduler/app.py`, add `SNAPSHOT_CYCLE_INTERVAL = int(os.getenv("SNAPSHOT_CYCLE_INTERVAL", "240"))` constant and `snapshot_counter = 0` initialization in `main()`
|
||
- **Validates: Bugfix 2.6, 2.7**
|
||
|
||
- [x] 7. Implement maybe_capture_daily_snapshots function — In `services/scheduler/app.py`, implement `maybe_capture_daily_snapshots(pool)` that checks time (after 16:30 ET), checks idempotency (no existing row for today), queries positions table for portfolio value/unrealized PnL, and inserts into `portfolio_snapshots` and `daily_risk_snapshots`
|
||
- **Validates: Bugfix 2.6, 2.7; Regression 3.10**
|
||
|
||
- [x] 8. Wire snapshot cycle into main loop — In `services/scheduler/app.py` main loop, add the counter increment and conditional call to `maybe_capture_daily_snapshots(pool)` after the validation counter block
|
||
- **Validates: Bugfix 2.6, 2.7**
|
||
|
||
- [x] 9. Add prediction price fallback — In `services/validation/prediction_snapshot.py`, after the primary market_snapshots price lookup returns NULL for `price_at_prediction`, add a fallback query to positions table: `SELECT current_price FROM positions WHERE ticker = $1 AND current_price IS NOT NULL LIMIT 1`
|
||
- **Validates: Bugfix 2.8; Regression 3.4**
|
||
|
||
- [x] 10. Scale down lake-publisher — In `infra/helm/stonks-oracle/values.yaml`, change the lake-publisher `replicas` from `1` to `0`
|
||
- **Validates: Bugfix 2.9; Regression 3.5**
|
||
|
||
- [x] 11. Extend _INSERT_ORDER SQL — In `services/adapters/broker_service.py`, extend `_INSERT_ORDER` SQL to include `rejection_reason` and `rejected_at` as parameters $18 and $19, with COALESCE in the ON CONFLICT UPDATE clause to preserve existing values
|
||
- **Validates: Bugfix 2.10; Regression 3.6, 3.8**
|
||
|
||
- [x] 12. Update persist_order parameters — In `services/adapters/broker_service.py`, update `persist_order()` to compute `rejection_reason = resp.error if resp.status == OrderStatus.REJECTED else None` and `rejected_at = now if resp.status == OrderStatus.REJECTED else None`, passing them as the final two parameters in the execute call
|
||
- **Validates: Bugfix 2.10; Regression 3.6, 3.8**
|
||
|
||
- [x] 13. Lint and test — Run `.venv/bin/ruff check services/` and `.venv/bin/python -m pytest tests/ -x --tb=short -q` to verify no regressions
|
||
- **Validates: Regression 3.1–3.10**
|
||
|
||
## Task Dependency Graph
|
||
|
||
```json
|
||
{
|
||
"waves": [
|
||
{"tasks": [1, 2, 9, 10]},
|
||
{"tasks": [3, 6, 11]},
|
||
{"tasks": [4, 7, 12]},
|
||
{"tasks": [5, 8]},
|
||
{"tasks": [13]}
|
||
]
|
||
}
|
||
```
|
||
|
||
Tasks 1, 2, 9, 10 are fully independent. Tasks 3/6/11 set up constants needed by 4/7/12. Tasks 5/8 wire into the main loop after their functions exist. Task 13 validates everything last.
|
||
|
||
## Notes
|
||
|
||
- No database migrations required — all tables already exist with correct columns
|
||
- All scheduler changes use the existing counter-based periodic task pattern already established for cleanup, aggregation, and report tasks
|
||
- Lazy imports in `run_validation_cycle` avoid circular imports and keep scheduler startup fast
|
||
- The `maybe_capture_daily_snapshots` idempotency check prevents duplicate rows on scheduler restart
|