Celes Renata
ca712ad4a0
- Scheduler: lower stale threshold 240→30 min, batch limit 100→500, TTL 14400→3600 - Prediction snapshot: add 24h market_snapshots time-window fallback - Aggregation: add normalize_impact_scores() z-score normalization - Helm: signal-engine replicas → 0 (idle when dual pipeline disabled) - Quality gate: max_snapshot_age_hours 24→48 - Add backfill script for NULL price_at_prediction snapshots - Add PBT bug condition and preservation tests (14 tests)
5.4 KiB
Bugfix Requirements Document
Introduction
Five operational bugs in the stonks-beta deployment degrade pipeline health: 1,809 documents stuck in parsed status due to recovery batch limits, 26.5% of prediction snapshots missing prices due to incomplete fallback chains, 64% sell bias from uncalibrated NuExtract3 sentiment outputs, idle signal-engine consuming resources while doing nothing, and a quality gate stuck in paper-only mode due to an overly strict staleness threshold interacting with the NULL price problem.
Bug Analysis
Current Behavior (Defect)
1.1 WHEN the recover_stale_documents task runs with 1,809+ documents stuck in parsed status THEN the system only processes 100 per cycle (every ~5 minutes), requiring 90+ cycles (~7.5 hours) to clear the backlog while new documents may continue accumulating
1.2 WHEN a prediction snapshot is created for a ticker without an open position AND without recent market_snapshots data THEN the system stores NULL in price_at_prediction because the fallback chain stops at the positions table (26.5% of snapshots affected — 33,324 of 125,590)
1.3 WHEN the outcome evaluator encounters a prediction snapshot with NULL price_at_prediction THEN the system skips the snapshot entirely, creating a validation blind spot where 26.5% of predictions are never evaluated
1.4 WHEN the aggregation pipeline processes NuExtract3 extraction outputs THEN the system passes raw impact_score and sentiment values directly into signal weighting without any distribution normalization, resulting in systematic negative bias producing 64% sell / 23% watch / 12% buy recommendations
1.5 WHEN the signal-engine pod starts with dual_pipeline_enabled=False THEN the system enters an infinite sleep loop consuming CPU (100m request / 500m limit) and memory (128Mi request / 256Mi limit) while producing zero signal evaluations
1.6 WHEN the quality gate checks model_metric_snapshots freshness with a 24-hour staleness threshold AND the validation cycle skips all predictions due to NULL prices (Bug 1.2/1.3) THEN the system permanently defaults to paper-only mode because no fresh metric snapshots are ever generated
Expected Behavior (Correct)
2.1 WHEN the scheduler detects more than 100 documents stuck in parsed status older than the threshold THEN the system SHALL increase the batch limit for recovery processing (up to 500 per cycle) and provide a one-time management command to bulk-recover the existing backlog without waiting for periodic sweeps
2.2 WHEN a prediction snapshot is created and no price is available from market_snapshots (exact time) or positions table THEN the system SHALL query market_snapshots with a wider time window (last 24 hours of bar data for the ticker) as an additional fallback before accepting NULL
2.3 WHEN backfilling existing prediction snapshots with NULL price_at_prediction THEN the system SHALL use the extended fallback chain (market_snapshots within 24h of generated_at, then positions) to populate prices retroactively via a migration script
2.4 WHEN the aggregation pipeline computes signal weights from impact records THEN the system SHALL apply z-score normalization to impact_score values relative to the rolling 7-day distribution of impact records for the same ticker, preventing systematic model bias from dominating the directional signal
2.5 WHEN the signal-engine deployment is not ready for production use (dual_pipeline_enabled=False) THEN the system SHALL be scaled to 0 replicas in the Helm values files (beta, paper, live) to eliminate wasted CPU, memory, and any GPU time-slice allocations
2.6 WHEN the quality gate evaluates metric snapshot freshness during the bootstrapping period THEN the system SHALL use a 48-hour staleness threshold (instead of 24h) to tolerate gaps while the validation cycle ramps up after Bug 1.2/1.3 are fixed
Unchanged Behavior (Regression Prevention)
3.1 WHEN documents enter parsed status and are processed within the normal threshold window (< 240 minutes) THEN the system SHALL CONTINUE TO leave them for the extraction queue consumer without interference from the recovery task
3.2 WHEN a prediction snapshot is created and market_snapshots contains a recent bar for the ticker THEN the system SHALL CONTINUE TO use the primary market_snapshots close price without invoking any fallback
3.3 WHEN the aggregation pipeline processes tickers with balanced sentiment distributions (equal bullish/bearish evidence) THEN the system SHALL CONTINUE TO produce neutral/mixed recommendations without artificial skew from the normalization step
3.4 WHEN the signal-engine is re-enabled in the future (dual_pipeline_enabled=True with replicas > 0) THEN the system SHALL CONTINUE TO function correctly with its existing queue-based architecture and configuration loading
3.5 WHEN the quality gate evaluates a metric snapshot that is less than 48 hours old and meets all threshold criteria THEN the system SHALL CONTINUE TO promote recommendations to live_eligible mode per existing threshold logic
3.6 WHEN the outcome evaluator processes prediction snapshots with valid (non-NULL) prices THEN the system SHALL CONTINUE TO evaluate them normally and produce prediction_outcomes records
3.7 WHEN the retry_failed_extractions task handles documents in extraction_failed status THEN the system SHALL CONTINUE TO process them on the existing cadence and logic without interference from the parsed-document recovery changes