fix: pipeline health — stuck docs, price fallback, sentiment normalization, signal-engine scale, quality gate
- Scheduler: lower stale threshold 240→30 min, batch limit 100→500, TTL 14400→3600 - Prediction snapshot: add 24h market_snapshots time-window fallback - Aggregation: add normalize_impact_scores() z-score normalization - Helm: signal-engine replicas → 0 (idle when dual pipeline disabled) - Quality gate: max_snapshot_age_hours 24→48 - Add backfill script for NULL price_at_prediction snapshots - Add PBT bug condition and preservation tests (14 tests)
This commit is contained in:
Celes Renata
@@ -0,0 +1 @@
|
||||
{"specId": "f5d99301-94ef-4dc2-8ba4-ccefeee7ecba", "workflowType": "requirements-first", "specType": "bugfix"}
|
||||
@@ -0,0 +1,51 @@
|
||||
# Bugfix Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
Five operational bugs in the stonks-beta deployment degrade pipeline health: 1,809 documents stuck in `parsed` status due to recovery batch limits, 26.5% of prediction snapshots missing prices due to incomplete fallback chains, 64% sell bias from uncalibrated NuExtract3 sentiment outputs, idle signal-engine consuming resources while doing nothing, and a quality gate stuck in paper-only mode due to an overly strict staleness threshold interacting with the NULL price problem.
|
||||
|
||||
## Bug Analysis
|
||||
|
||||
### Current Behavior (Defect)
|
||||
|
||||
1.1 WHEN the `recover_stale_documents` task runs with 1,809+ documents stuck in `parsed` status THEN the system only processes 100 per cycle (every ~5 minutes), requiring 90+ cycles (~7.5 hours) to clear the backlog while new documents may continue accumulating
|
||||
|
||||
1.2 WHEN a prediction snapshot is created for a ticker without an open position AND without recent market_snapshots data THEN the system stores NULL in `price_at_prediction` because the fallback chain stops at the positions table (26.5% of snapshots affected — 33,324 of 125,590)
|
||||
|
||||
1.3 WHEN the outcome evaluator encounters a prediction snapshot with NULL `price_at_prediction` THEN the system skips the snapshot entirely, creating a validation blind spot where 26.5% of predictions are never evaluated
|
||||
|
||||
1.4 WHEN the aggregation pipeline processes NuExtract3 extraction outputs THEN the system passes raw `impact_score` and `sentiment` values directly into signal weighting without any distribution normalization, resulting in systematic negative bias producing 64% sell / 23% watch / 12% buy recommendations
|
||||
|
||||
1.5 WHEN the signal-engine pod starts with `dual_pipeline_enabled=False` THEN the system enters an infinite sleep loop consuming CPU (100m request / 500m limit) and memory (128Mi request / 256Mi limit) while producing zero signal evaluations
|
||||
|
||||
1.6 WHEN the quality gate checks `model_metric_snapshots` freshness with a 24-hour staleness threshold AND the validation cycle skips all predictions due to NULL prices (Bug 1.2/1.3) THEN the system permanently defaults to paper-only mode because no fresh metric snapshots are ever generated
|
||||
|
||||
### Expected Behavior (Correct)
|
||||
|
||||
2.1 WHEN the scheduler detects more than 100 documents stuck in `parsed` status older than the threshold THEN the system SHALL increase the batch limit for recovery processing (up to 500 per cycle) and provide a one-time management command to bulk-recover the existing backlog without waiting for periodic sweeps
|
||||
|
||||
2.2 WHEN a prediction snapshot is created and no price is available from market_snapshots (exact time) or positions table THEN the system SHALL query `market_snapshots` with a wider time window (last 24 hours of bar data for the ticker) as an additional fallback before accepting NULL
|
||||
|
||||
2.3 WHEN backfilling existing prediction snapshots with NULL `price_at_prediction` THEN the system SHALL use the extended fallback chain (market_snapshots within 24h of `generated_at`, then positions) to populate prices retroactively via a migration script
|
||||
|
||||
2.4 WHEN the aggregation pipeline computes signal weights from impact records THEN the system SHALL apply z-score normalization to `impact_score` values relative to the rolling 7-day distribution of impact records for the same ticker, preventing systematic model bias from dominating the directional signal
|
||||
|
||||
2.5 WHEN the signal-engine deployment is not ready for production use (`dual_pipeline_enabled=False`) THEN the system SHALL be scaled to 0 replicas in the Helm values files (beta, paper, live) to eliminate wasted CPU, memory, and any GPU time-slice allocations
|
||||
|
||||
2.6 WHEN the quality gate evaluates metric snapshot freshness during the bootstrapping period THEN the system SHALL use a 48-hour staleness threshold (instead of 24h) to tolerate gaps while the validation cycle ramps up after Bug 1.2/1.3 are fixed
|
||||
|
||||
### Unchanged Behavior (Regression Prevention)
|
||||
|
||||
3.1 WHEN documents enter `parsed` status and are processed within the normal threshold window (< 240 minutes) THEN the system SHALL CONTINUE TO leave them for the extraction queue consumer without interference from the recovery task
|
||||
|
||||
3.2 WHEN a prediction snapshot is created and market_snapshots contains a recent bar for the ticker THEN the system SHALL CONTINUE TO use the primary `market_snapshots` close price without invoking any fallback
|
||||
|
||||
3.3 WHEN the aggregation pipeline processes tickers with balanced sentiment distributions (equal bullish/bearish evidence) THEN the system SHALL CONTINUE TO produce neutral/mixed recommendations without artificial skew from the normalization step
|
||||
|
||||
3.4 WHEN the signal-engine is re-enabled in the future (dual_pipeline_enabled=True with replicas > 0) THEN the system SHALL CONTINUE TO function correctly with its existing queue-based architecture and configuration loading
|
||||
|
||||
3.5 WHEN the quality gate evaluates a metric snapshot that is less than 48 hours old and meets all threshold criteria THEN the system SHALL CONTINUE TO promote recommendations to live_eligible mode per existing threshold logic
|
||||
|
||||
3.6 WHEN the outcome evaluator processes prediction snapshots with valid (non-NULL) prices THEN the system SHALL CONTINUE TO evaluate them normally and produce prediction_outcomes records
|
||||
|
||||
3.7 WHEN the `retry_failed_extractions` task handles documents in `extraction_failed` status THEN the system SHALL CONTINUE TO process them on the existing cadence and logic without interference from the parsed-document recovery changes
|
||||
@@ -0,0 +1,320 @@
|
||||
# Pipeline Health Fixes — Bugfix Design
|
||||
|
||||
## Overview
|
||||
|
||||
Five operational bugs degrade stonks-beta pipeline health. This design formalizes the bug conditions, expected fixes, and validation strategy for each:
|
||||
|
||||
1. **Stuck Parsed Docs** — `recover_stale_documents()` batch limit of 100 is too low for 1,809 stuck documents; increase to 500 and lower the stale threshold to 30 minutes.
|
||||
2. **Extended Price Fallback** — Prediction snapshots missing prices (26.5%) because the fallback chain stops at `positions`; add a third fallback querying `market_snapshots` within 24h.
|
||||
3. **Sentiment Z-Score Normalization** — Raw NuExtract3 `impact_score` values produce 64% sell bias; normalize using 7-day rolling z-scores per ticker before signal weighting.
|
||||
4. **Signal Engine Scale Down** — Idle signal-engine pods consume resources; set replicas to 0 in all Helm values files.
|
||||
5. **Quality Gate Threshold** — 24h staleness threshold permanently locks quality gate to paper-only; relax to 48h.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **Bug_Condition (C)**: The specific conditions under which each bug manifests
|
||||
- **Property (P)**: The desired correct behavior after the fix is applied
|
||||
- **Preservation**: Existing behavior that must remain unchanged after the fix
|
||||
- **`recover_stale_documents()`**: Function in `services/scheduler/app.py` that re-enqueues documents stuck in `parsed` status
|
||||
- **`STALE_PARSED_THRESHOLD_MINUTES`**: Constant (currently 240) controlling how long a document must be stuck before recovery
|
||||
- **`fetch_latest_close_price()`**: Function in `services/validation/prediction_snapshot.py` that queries `market_snapshots` for the most recent bar
|
||||
- **`compute_signal_weight()`**: Function in `services/aggregation/scoring.py` that computes combined signal weight from recency, credibility, novelty, confidence, and impact
|
||||
- **`QualityGateConfig.max_snapshot_age_hours`**: Threshold in `services/trading/model_quality_gate.py` controlling when the quality gate defaults to paper-only
|
||||
|
||||
## Bug Details
|
||||
|
||||
### Bug Condition
|
||||
|
||||
The pipeline health degradation manifests across five independent conditions:
|
||||
|
||||
**Formal Specification:**
|
||||
```
|
||||
FUNCTION isBugCondition(input)
|
||||
INPUT: input of type PipelineState
|
||||
OUTPUT: boolean
|
||||
|
||||
-- Bug 1: Parsed docs stuck beyond batch capacity
|
||||
RETURN (input.stuckParsedDocCount > 100
|
||||
AND input.recoveryBatchLimit == 100
|
||||
AND input.docStaleMinutes >= 240)
|
||||
-- Bug 2: Price fallback chain incomplete
|
||||
OR (input.tickerPrice IS NULL
|
||||
AND input.positionPrice IS NULL
|
||||
AND input.marketSnapshotWithin24h IS NOT NULL)
|
||||
-- Bug 3: Raw impact scores without normalization
|
||||
OR (input.impactScoreUsedRaw == TRUE
|
||||
AND input.ticker7dStddev > 0)
|
||||
-- Bug 4: Signal engine running idle
|
||||
OR (input.signalEngineReplicas > 0
|
||||
AND input.dualPipelineEnabled == FALSE)
|
||||
-- Bug 5: Quality gate threshold too strict
|
||||
OR (input.snapshotAgeHours > 24
|
||||
AND input.snapshotAgeHours <= 48
|
||||
AND input.maxSnapshotAgeConfig == 24)
|
||||
END FUNCTION
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
- **Bug 1**: 1,809 documents in `parsed` status older than 4 hours. At 100/cycle every 5 minutes, clearing takes 90+ cycles (~7.5h). With 500/batch, it takes 4 cycles (~20 min).
|
||||
- **Bug 2**: Ticker PLTR has no open position and `fetch_latest_close_price` returns NULL, but `market_snapshots` has a bar from 3 hours ago that could serve as price.
|
||||
- **Bug 3**: NuExtract3 outputs `impact_score` values clustered around -0.3 to -0.1 for a ticker. Without normalization, `weighted_sentiment_average()` systematically produces negative signals → 64% sell recommendations.
|
||||
- **Bug 4**: signal-engine pod starts, detects `dual_pipeline_enabled=False`, enters infinite sleep loop consuming 100m CPU request / 128Mi memory request.
|
||||
- **Bug 5**: Quality gate reads `model_metric_snapshots`, finds the most recent is 26h old (because validation skips NULL-price predictions), fails staleness check, forces paper-only mode permanently.
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
### Preservation Requirements
|
||||
|
||||
**Unchanged Behaviors:**
|
||||
- Documents entering `parsed` status and processed within the normal threshold window (< 30 min after fix) are left alone for the extraction queue consumer
|
||||
- Primary price lookup via `fetch_latest_close_price()` (exact time match from `market_snapshots`) continues as the first-choice price source
|
||||
- Tickers with balanced sentiment distributions continue to produce neutral/mixed recommendations without artificial skew
|
||||
- Signal-engine's queue-based architecture and configuration loading remain functional when re-enabled with replicas > 0
|
||||
- Quality gate threshold logic for snapshots younger than 48h and meeting all criteria continues to promote to `live_eligible`
|
||||
- Outcome evaluator continues to process prediction snapshots with valid (non-NULL) prices normally
|
||||
- `retry_failed_extractions` task continues on its existing cadence without interference
|
||||
|
||||
**Scope:**
|
||||
All inputs that do NOT match the bug conditions above should be completely unaffected by these fixes. The fixes are additive (new fallback path, wider batch, normalization layer) or config-only (replica count, threshold constant).
|
||||
|
||||
## Hypothesized Root Cause
|
||||
|
||||
### Bug 1: Stuck Parsed Docs
|
||||
- **Batch limit too small**: The `LIMIT 100` in the SQL query caps recovery throughput at 100 docs per scheduler cycle (~5 min). When a Redis crash orphans thousands of documents, the recovery rate cannot keep up with the backlog.
|
||||
- **Threshold too conservative**: `STALE_PARSED_THRESHOLD_MINUTES = 240` (4 hours) means documents must be stuck for 4 hours before recovery kicks in. A 30-minute threshold would catch orphans much faster.
|
||||
|
||||
### Bug 2: Incomplete Fallback Chain
|
||||
- **Missing time-window query**: `fetch_latest_close_price()` only checks `market_snapshots` for an exact timestamp match. When market data ingestion is delayed or the prediction happens outside market hours, no exact match exists.
|
||||
- **Positions-only fallback**: The positions table fallback only works for tickers with an active position. 26.5% of snapshots are for tickers without positions.
|
||||
|
||||
### Bug 3: Uncalibrated Impact Scores
|
||||
- **No distribution normalization**: NuExtract3 model outputs are passed raw into `compute_signal_weight()` via `impact_score` parameter. The model has a systematic negative bias in its output distribution that is not corrected.
|
||||
- **Per-ticker variance ignored**: Different tickers receive different volume/types of news, producing different impact_score distributions. A global normalization would be insufficient.
|
||||
|
||||
### Bug 4: Idle Signal Engine
|
||||
- **Replicas set to 1 by default**: `values.yaml` defines `signalEngine.replicas: 1` regardless of whether the dual pipeline feature is enabled. The pod starts, detects the feature is off, and sleeps forever.
|
||||
|
||||
### Bug 5: Overly Strict Staleness
|
||||
- **24h threshold too tight during bootstrapping**: The `max_snapshot_age_hours = 24` default assumes the validation cycle runs frequently. When Bug 2/3 cause most predictions to be skipped, metric snapshots aren't generated, and the 24h window expires.
|
||||
|
||||
## Correctness Properties
|
||||
|
||||
Property 1: Bug Condition — Stuck Parsed Docs Recovery
|
||||
|
||||
_For any_ set of documents stuck in `parsed` status longer than 30 minutes, the fixed `recover_stale_documents()` function SHALL process up to 500 documents per cycle, reducing backlog clearance time by 5x compared to the previous 100-document limit.
|
||||
|
||||
**Validates: Requirements 2.1**
|
||||
|
||||
Property 2: Bug Condition — Extended Price Fallback
|
||||
|
||||
_For any_ prediction snapshot where `fetch_latest_close_price()` returns NULL and the `positions` table has no price, but `market_snapshots` contains a bar for the ticker within 24 hours of the prediction time, the fixed code SHALL use that bar's close price as `price_at_prediction`.
|
||||
|
||||
**Validates: Requirements 2.2, 2.3**
|
||||
|
||||
Property 3: Bug Condition — Sentiment Z-Score Normalization
|
||||
|
||||
_For any_ set of `document_impact_records` for a ticker, the fixed aggregation pipeline SHALL normalize `impact_score` values using the 7-day rolling mean and standard deviation for that ticker before passing them into signal weight computation, preventing systematic model bias.
|
||||
|
||||
**Validates: Requirements 2.4**
|
||||
|
||||
Property 4: Bug Condition — Signal Engine Scale Down
|
||||
|
||||
_For any_ Helm deployment where `dual_pipeline_enabled=False`, the fixed Helm values SHALL specify `signalEngine.replicas: 0`, preventing the pod from being scheduled and consuming resources.
|
||||
|
||||
**Validates: Requirements 2.5**
|
||||
|
||||
Property 5: Bug Condition — Quality Gate Threshold
|
||||
|
||||
_For any_ model metric snapshot that is between 24h and 48h old, the fixed quality gate SHALL NOT reject it as stale, allowing the system to remain in non-paper mode during the bootstrapping period.
|
||||
|
||||
**Validates: Requirements 2.6**
|
||||
|
||||
Property 6: Preservation — Normal Document Processing
|
||||
|
||||
_For any_ document that enters `parsed` status and is processed within 30 minutes, the fixed `recover_stale_documents()` function SHALL NOT interfere with normal extraction queue processing, preserving the existing pipeline flow.
|
||||
|
||||
**Validates: Requirements 3.1, 3.7**
|
||||
|
||||
Property 7: Preservation — Primary Price Path
|
||||
|
||||
_For any_ prediction snapshot where `fetch_latest_close_price()` returns a valid price, the fixed code SHALL use that price directly without invoking any fallback, preserving the primary price lookup behavior.
|
||||
|
||||
**Validates: Requirements 3.2**
|
||||
|
||||
Property 8: Preservation — Balanced Sentiment
|
||||
|
||||
_For any_ ticker with a balanced sentiment distribution (equal bullish/bearish evidence), the z-score normalization SHALL produce values centered around 0, preserving neutral/mixed recommendation output without artificial skew.
|
||||
|
||||
**Validates: Requirements 3.3**
|
||||
|
||||
Property 9: Preservation — Quality Gate Valid Snapshots
|
||||
|
||||
_For any_ model metric snapshot younger than 48h that meets all threshold criteria, the fixed quality gate SHALL continue to promote recommendations to `live_eligible` mode per existing logic.
|
||||
|
||||
**Validates: Requirements 3.5, 3.6**
|
||||
|
||||
## Fix Implementation
|
||||
|
||||
### Changes Required
|
||||
|
||||
**Bug 1: Stuck Parsed Docs Recovery**
|
||||
|
||||
**File**: `services/scheduler/app.py`
|
||||
|
||||
**Function**: `recover_stale_documents()`
|
||||
|
||||
**Specific Changes**:
|
||||
1. **Lower stale threshold**: Change `STALE_PARSED_THRESHOLD_MINUTES` from `240` to `30` — documents stuck longer than 30 minutes are likely orphaned
|
||||
2. **Increase batch limit**: Change `LIMIT 100` to `LIMIT 500` in the SQL query
|
||||
3. **Update enqueued TTL**: Change `_ENQUEUED_TTL` from `14400` (4h) to `3600` (1h) to match the new threshold
|
||||
|
||||
---
|
||||
|
||||
**Bug 2: Extended Price Fallback**
|
||||
|
||||
**File**: `services/validation/prediction_snapshot.py`
|
||||
|
||||
**Function**: `create_prediction_snapshot()`
|
||||
|
||||
**Specific Changes**:
|
||||
1. **Add market_snapshots time-window fallback**: After the positions fallback fails, query `market_snapshots` for the most recent bar within 24h of the current time for the ticker
|
||||
2. **SQL query**: `SELECT close FROM market_snapshots WHERE ticker = $1 AND timestamp >= NOW() - INTERVAL '24 hours' ORDER BY timestamp DESC LIMIT 1`
|
||||
3. **Log the fallback**: Add info-level logging when the extended fallback is used
|
||||
|
||||
**New File**: `scripts/backfill_snapshot_prices.py`
|
||||
|
||||
**Purpose**: One-time backfill script to populate `price_at_prediction` for existing NULL snapshots using the extended fallback chain.
|
||||
|
||||
**Approach**:
|
||||
1. Query all `prediction_snapshots` where `price_at_prediction IS NULL`
|
||||
2. For each, attempt: `market_snapshots` within 24h of `generated_at`, then `positions` table
|
||||
3. Update the row with the found price
|
||||
4. Report statistics (found via market_snapshots, found via positions, still NULL)
|
||||
|
||||
---
|
||||
|
||||
**Bug 3: Sentiment Z-Score Normalization**
|
||||
|
||||
**File**: `services/aggregation/worker.py` (or new helper in `services/aggregation/scoring.py`)
|
||||
|
||||
**Function**: New function `normalize_impact_scores()` called before `compute_signal_weight()`
|
||||
|
||||
**Specific Changes**:
|
||||
1. **Add normalization function**: Compute 7-day rolling mean and stddev of `impact_score` per ticker from `document_impact_records`
|
||||
2. **Formula**: `normalized = (raw - mean_7d) / max(stddev_7d, 0.1)` — the 0.1 floor prevents division by near-zero stddev for low-activity tickers
|
||||
3. **Integration point**: In the aggregation loop (around line 440 of worker.py), normalize `imp.impact_score` before passing to `compute_signal_weight()` and `WeightedSignal`
|
||||
4. **Fallback**: If fewer than 5 records exist in the 7-day window, use the raw score (insufficient data for meaningful normalization)
|
||||
5. **Query**: `SELECT AVG(impact_score) as mean, STDDEV(impact_score) as stddev FROM document_impact_records WHERE ticker = $1 AND created_at >= NOW() - INTERVAL '7 days'`
|
||||
|
||||
---
|
||||
|
||||
**Bug 4: Signal Engine Scale Down**
|
||||
|
||||
**Files**: `infra/helm/stonks-oracle/values.yaml`, `values-beta.yaml`, `values-paper.yaml`
|
||||
|
||||
**Specific Changes**:
|
||||
1. **values.yaml**: Change `signalEngine.replicas` from `1` to `0`
|
||||
2. **values-beta.yaml**: Add `signalEngine.replicas: 0` under `services:`
|
||||
3. **values-paper.yaml**: Add `signalEngine.replicas: 0` under `services:`
|
||||
|
||||
---
|
||||
|
||||
**Bug 5: Quality Gate Threshold**
|
||||
|
||||
**File**: `services/trading/model_quality_gate.py`
|
||||
|
||||
**Class**: `QualityGateConfig`
|
||||
|
||||
**Specific Changes**:
|
||||
1. **Change default**: `max_snapshot_age_hours: int = 48` (was 24)
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Validation Approach
|
||||
|
||||
The testing strategy follows a two-phase approach: first, surface counterexamples that demonstrate the bug on unfixed code, then verify the fix works correctly and preserves existing behavior.
|
||||
|
||||
### Exploratory Bug Condition Checking
|
||||
|
||||
**Goal**: Surface counterexamples that demonstrate the bugs BEFORE implementing the fixes. Confirm or refute the root cause analysis.
|
||||
|
||||
**Test Plan**: Write tests that exercise each bug condition on the unfixed code to observe failures.
|
||||
|
||||
**Test Cases**:
|
||||
1. **Batch Overflow Test**: Create 600 documents in `parsed` status older than threshold, run `recover_stale_documents()`, assert only 100 are processed (will demonstrate Bug 1)
|
||||
2. **Price Fallback Gap Test**: Call `create_prediction_snapshot()` for a ticker with no position and no exact market_snapshots match, assert `price_at_prediction` is NULL (will demonstrate Bug 2)
|
||||
3. **Sentiment Bias Test**: Generate 50 impact records with systematic negative bias (mean=-0.3, stddev=0.1), compute weighted signals, assert directional signal is negative (will demonstrate Bug 3)
|
||||
4. **Quality Gate Staleness Test**: Set most recent metric snapshot to 26h ago, evaluate quality gate, assert it fails (will demonstrate Bug 5)
|
||||
|
||||
**Expected Counterexamples**:
|
||||
- Bug 1: Only 100 of 600 documents recovered per cycle
|
||||
- Bug 2: `price_at_prediction` stored as NULL despite market data existing within 24h
|
||||
- Bug 3: Weighted sentiment average heavily negative despite mixed underlying events
|
||||
- Bug 5: Quality gate returns `passed=False` with reason containing "stale"
|
||||
|
||||
### Fix Checking
|
||||
|
||||
**Goal**: Verify that for all inputs where the bug condition holds, the fixed function produces the expected behavior.
|
||||
|
||||
**Pseudocode:**
|
||||
```
|
||||
FOR ALL input WHERE isBugCondition(input) DO
|
||||
result := fixedFunction(input)
|
||||
ASSERT expectedBehavior(result)
|
||||
END FOR
|
||||
```
|
||||
|
||||
**Per-bug fix checks:**
|
||||
- Bug 1: `recover_stale_documents()` processes up to 500 docs with 30-min threshold
|
||||
- Bug 2: Extended fallback returns a price when `market_snapshots` has data within 24h
|
||||
- Bug 3: Normalized impact scores have mean ≈ 0 and stddev ≈ 1 for active tickers
|
||||
- Bug 4: `kubectl get pods` shows 0 signal-engine pods
|
||||
- Bug 5: Quality gate passes for snapshots 24–48h old that meet metric thresholds
|
||||
|
||||
### Preservation Checking
|
||||
|
||||
**Goal**: Verify that for all inputs where the bug condition does NOT hold, the fixed function produces the same result as the original function.
|
||||
|
||||
**Pseudocode:**
|
||||
```
|
||||
FOR ALL input WHERE NOT isBugCondition(input) DO
|
||||
ASSERT originalFunction(input) = fixedFunction(input)
|
||||
END FOR
|
||||
```
|
||||
|
||||
**Testing Approach**: Property-based testing is recommended for preservation checking because:
|
||||
- It generates many test cases automatically across the input domain
|
||||
- It catches edge cases that manual unit tests might miss
|
||||
- It provides strong guarantees that behavior is unchanged for all non-buggy inputs
|
||||
|
||||
**Test Plan**: Observe behavior on UNFIXED code first for normal inputs, then write property-based tests capturing that behavior.
|
||||
|
||||
**Test Cases**:
|
||||
1. **Normal Doc Processing Preservation**: Documents < 30 min old are never touched by recovery
|
||||
2. **Primary Price Preservation**: When `fetch_latest_close_price()` succeeds, no fallback is invoked
|
||||
3. **Balanced Sentiment Preservation**: Tickers with symmetric impact_score distributions produce neutral signals after normalization
|
||||
4. **Quality Gate Normal Preservation**: Snapshots < 48h old and meeting thresholds still pass
|
||||
5. **Failed Extraction Preservation**: `retry_failed_extractions()` behavior unchanged
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- Test `recover_stale_documents()` with various document counts (0, 50, 500, 1000)
|
||||
- Test extended price fallback with market_snapshots at various time offsets (1h, 12h, 23h, 25h)
|
||||
- Test z-score normalization with known distributions (mean=0, mean=-0.5, stddev=0, stddev=0.05)
|
||||
- Test quality gate with snapshot ages at boundary (23h, 24h, 47h, 48h, 49h)
|
||||
- Test backfill script with mixed NULL/non-NULL snapshots
|
||||
|
||||
### Property-Based Tests
|
||||
|
||||
- Generate random document ages and counts, verify recovery processes correct subset (> 30 min old, up to 500)
|
||||
- Generate random ticker price scenarios, verify fallback chain ordering is preserved (primary → positions → market_snapshots_24h → NULL)
|
||||
- Generate random impact_score distributions per ticker, verify normalized output has bounded variance and zero-centered mean
|
||||
- Generate random snapshot ages, verify quality gate accepts [0, 48h) and rejects [48h, ∞)
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- End-to-end: create documents in `parsed` status, run scheduler cycle, verify extraction queue populated
|
||||
- End-to-end: create prediction snapshot for ticker without position, verify price populated from market_snapshots
|
||||
- End-to-end: run full aggregation cycle with biased NuExtract3 outputs, verify recommendation direction is not systematically biased
|
||||
- Helm template render: verify signal-engine deployment has 0 replicas in all value files
|
||||
@@ -0,0 +1,174 @@
|
||||
# Implementation Plan
|
||||
|
||||
## Overview
|
||||
|
||||
Bugfix implementation for five pipeline health issues: stuck parsed docs, missing price fallback, uncalibrated sentiment scores, idle signal-engine pods, and overly strict quality gate threshold. Tasks follow the exploratory bugfix workflow: explore bugs via tests, preserve existing behavior, implement fixes, validate.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] 1. Write bug condition exploration test
|
||||
- **Property 1: Bug Condition** - Pipeline Health Degradation
|
||||
- **CRITICAL**: This test MUST FAIL on unfixed code - failure confirms the bugs exist
|
||||
- **DO NOT attempt to fix the test or the code when it fails**
|
||||
- **NOTE**: This test encodes the expected behavior - it will validate the fix when it passes after implementation
|
||||
- **GOAL**: Surface counterexamples that demonstrate all five bugs exist
|
||||
- **Scoped PBT Approach**: Scope properties to the concrete failing cases for each bug condition
|
||||
- Test file: `tests/test_pbt_pipeline_health_bug_condition.py`
|
||||
- **Bug 1 - Batch Overflow**: Create 600 documents in `parsed` status older than 30 min, run `recover_stale_documents()`, assert up to 500 are recovered per cycle (will FAIL on unfixed code which caps at 100)
|
||||
- **Bug 2 - Price Fallback Gap**: Call `create_prediction_snapshot()` for ticker with no position and no exact market_snapshots match but data within 24h exists, assert `price_at_prediction` is NOT NULL (will FAIL on unfixed code which returns NULL)
|
||||
- **Bug 3 - Sentiment Bias**: Generate 50 impact records with systematic negative bias (mean=-0.3, stddev=0.1), compute weighted signals via aggregation, assert normalized output is zero-centered (will FAIL on unfixed code which passes raw scores)
|
||||
- **Bug 5 - Quality Gate Staleness**: Set most recent metric snapshot to 26h ago, evaluate quality gate, assert it passes (will FAIL on unfixed code which rejects at 24h)
|
||||
- Run tests on UNFIXED code
|
||||
- **EXPECTED OUTCOME**: Tests FAIL (this is correct - it proves the bugs exist)
|
||||
- Document counterexamples: batch capped at 100, price stored as NULL, sentiment heavily negative, quality gate returns `passed=False`
|
||||
- Mark task complete when tests are written, run, and failures are documented
|
||||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.6_
|
||||
|
||||
- [x] 2. Write preservation property tests (BEFORE implementing fix)
|
||||
- **Property 2: Preservation** - Pipeline Behavior Unchanged for Non-Bug Inputs
|
||||
- **IMPORTANT**: Follow observation-first methodology
|
||||
- Test file: `tests/test_pbt_pipeline_health_preservation.py`
|
||||
- **Normal Doc Processing**: Observe that documents < 30 min old are never touched by `recover_stale_documents()` on unfixed code. Write property: for all documents with age < 30 min, recovery task does NOT enqueue them.
|
||||
- **Primary Price Path**: Observe that when `fetch_latest_close_price()` returns a valid price, no fallback is invoked. Write property: for all tickers where primary price exists, result equals primary price.
|
||||
- **Balanced Sentiment**: Observe that tickers with symmetric impact_score distributions (mean ≈ 0) produce neutral signals. Write property: for all impact_score sets with mean ≈ 0, normalized output remains centered around 0.
|
||||
- **Quality Gate Normal**: Observe that snapshots < 48h old meeting thresholds pass the quality gate. Write property: for all snapshot ages in [0, 48h) meeting metric criteria, quality gate returns `passed=True`.
|
||||
- **Failed Extraction Independence**: Observe `retry_failed_extractions()` behavior is unaffected. Write property: for all documents in `extraction_failed` status, retry logic unchanged.
|
||||
- Run tests on UNFIXED code
|
||||
- **EXPECTED OUTCOME**: Tests PASS (this confirms baseline behavior to preserve)
|
||||
- Mark task complete when tests are written, run, and passing on unfixed code
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.5, 3.6, 3.7_
|
||||
|
||||
- [x] 3. Fix: Signal Engine Scale Down (Helm values)
|
||||
|
||||
- [x] 3.1 Set signal-engine replicas to 0 in all Helm values files
|
||||
- In `infra/helm/stonks-oracle/values.yaml`: change `signalEngine.replicas` from `1` to `0`
|
||||
- In `infra/helm/stonks-oracle/values-beta.yaml`: add/set `signalEngine.replicas: 0` under `services:`
|
||||
- In `infra/helm/stonks-oracle/values-paper.yaml`: add/set `signalEngine.replicas: 0` under `services:`
|
||||
- _Bug_Condition: input.signalEngineReplicas > 0 AND input.dualPipelineEnabled == FALSE_
|
||||
- _Expected_Behavior: signalEngine.replicas == 0 when dual pipeline disabled_
|
||||
- _Preservation: Signal-engine architecture remains functional when re-enabled with replicas > 0_
|
||||
- _Requirements: 2.5, 3.4_
|
||||
|
||||
- [x] 4. Fix: Quality Gate Threshold Relaxation
|
||||
|
||||
- [x] 4.1 Change max_snapshot_age_hours default from 24 to 48
|
||||
- File: `services/trading/model_quality_gate.py`
|
||||
- In `QualityGateConfig` class, change `max_snapshot_age_hours: int = 24` to `max_snapshot_age_hours: int = 48`
|
||||
- _Bug_Condition: input.snapshotAgeHours > 24 AND input.snapshotAgeHours <= 48 AND input.maxSnapshotAgeConfig == 24_
|
||||
- _Expected_Behavior: Quality gate accepts snapshots up to 48h old_
|
||||
- _Preservation: Snapshots < 48h meeting criteria continue to promote to live_eligible_
|
||||
- _Requirements: 2.6, 3.5_
|
||||
|
||||
- [x] 5. Fix: Stuck Parsed Docs Recovery
|
||||
|
||||
- [x] 5.1 Lower STALE_PARSED_THRESHOLD_MINUTES from 240 to 30
|
||||
- File: `services/scheduler/app.py`
|
||||
- Change constant: `STALE_PARSED_THRESHOLD_MINUTES = 30`
|
||||
- Documents stuck longer than 30 minutes are likely orphaned
|
||||
- _Requirements: 2.1_
|
||||
|
||||
- [x] 5.2 Increase recovery batch LIMIT from 100 to 500
|
||||
- File: `services/scheduler/app.py`
|
||||
- In `recover_stale_documents()` SQL query, change `LIMIT 100` to `LIMIT 500`
|
||||
- _Requirements: 2.1_
|
||||
|
||||
- [x] 5.3 Update _ENQUEUED_TTL from 14400 to 3600
|
||||
- File: `services/scheduler/app.py`
|
||||
- Change `_ENQUEUED_TTL = 3600` (1 hour, matching the new recovery cadence)
|
||||
- _Bug_Condition: input.stuckParsedDocCount > 100 AND input.recoveryBatchLimit == 100 AND input.docStaleMinutes >= 240_
|
||||
- _Expected_Behavior: Recovery processes up to 500 docs per cycle with 30-min threshold_
|
||||
- _Preservation: Documents < 30 min old left alone for extraction queue consumer_
|
||||
- _Requirements: 2.1, 3.1, 3.7_
|
||||
|
||||
- [x] 6. Fix: Extended Price Fallback
|
||||
|
||||
- [x] 6.1 Add market_snapshots 24h time-window fallback to create_prediction_snapshot()
|
||||
- File: `services/validation/prediction_snapshot.py`
|
||||
- After positions fallback fails, query: `SELECT close FROM market_snapshots WHERE ticker = $1 AND timestamp >= NOW() - INTERVAL '24 hours' ORDER BY timestamp DESC LIMIT 1`
|
||||
- Add info-level logging when extended fallback is used
|
||||
- _Bug_Condition: input.tickerPrice IS NULL AND input.positionPrice IS NULL AND input.marketSnapshotWithin24h IS NOT NULL_
|
||||
- _Expected_Behavior: Use market_snapshots bar close price as price_at_prediction_
|
||||
- _Preservation: Primary fetch_latest_close_price() path unchanged when it returns a valid price_
|
||||
- _Requirements: 2.2, 3.2_
|
||||
|
||||
- [x] 6.2 Create backfill script scripts/backfill_snapshot_prices.py
|
||||
- Query all `prediction_snapshots` where `price_at_prediction IS NULL`
|
||||
- For each, attempt: `market_snapshots` within 24h of `generated_at`, then `positions` table
|
||||
- Update row with found price
|
||||
- Report statistics: found via market_snapshots, found via positions, still NULL
|
||||
- _Requirements: 2.3_
|
||||
|
||||
- [x] 7. Fix: Sentiment Z-Score Normalization
|
||||
|
||||
- [x] 7.1 Add normalize_impact_scores() function
|
||||
- File: `services/aggregation/scoring.py` (new helper function)
|
||||
- Query 7-day mean and stddev per ticker: `SELECT AVG(impact_score) as mean, STDDEV(impact_score) as stddev FROM document_impact_records WHERE ticker = $1 AND created_at >= NOW() - INTERVAL '7 days'`
|
||||
- Formula: `normalized = (raw - mean_7d) / max(stddev_7d, 0.1)`
|
||||
- Fallback: if fewer than 5 records in 7-day window, return raw score unchanged
|
||||
- The 0.1 floor prevents division by near-zero stddev for low-activity tickers
|
||||
- _Requirements: 2.4_
|
||||
|
||||
- [x] 7.2 Integrate normalization into aggregation loop
|
||||
- File: `services/aggregation/worker.py`
|
||||
- Before `compute_signal_weight()` call (around line 440), normalize `imp.impact_score` via `normalize_impact_scores()`
|
||||
- Pass normalized value into `compute_signal_weight()` and `WeightedSignal`
|
||||
- _Bug_Condition: input.impactScoreUsedRaw == TRUE AND input.ticker7dStddev > 0_
|
||||
- _Expected_Behavior: Normalized impact scores with mean ≈ 0, stddev ≈ 1 for active tickers_
|
||||
- _Preservation: Tickers with balanced distributions continue to produce neutral signals_
|
||||
- _Requirements: 2.4, 3.3_
|
||||
|
||||
- [x] 8. Verify fixes pass all tests
|
||||
|
||||
- [x] 8.1 Verify bug condition exploration test now passes
|
||||
- **Property 1: Expected Behavior** - Pipeline Health Bugs Resolved
|
||||
- **IMPORTANT**: Re-run the SAME test from task 1 - do NOT write a new test
|
||||
- The test from task 1 encodes the expected behavior for all five bugs
|
||||
- Run `tests/test_pbt_pipeline_health_bug_condition.py`
|
||||
- **EXPECTED OUTCOME**: Test PASSES (confirms bugs are fixed)
|
||||
- _Requirements: 2.1, 2.2, 2.4, 2.6_
|
||||
|
||||
- [x] 8.2 Verify preservation tests still pass
|
||||
- **Property 2: Preservation** - Pipeline Behavior Unchanged for Non-Bug Inputs
|
||||
- **IMPORTANT**: Re-run the SAME tests from task 2 - do NOT write new tests
|
||||
- Run `tests/test_pbt_pipeline_health_preservation.py`
|
||||
- **EXPECTED OUTCOME**: Tests PASS (confirms no regressions)
|
||||
- Confirm all preservation properties still hold after fixes
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.5, 3.6, 3.7_
|
||||
|
||||
- [x] 9. Lint and final validation
|
||||
- Run `.venv/bin/ruff check services/` and fix any lint errors
|
||||
- Run `.venv/bin/python -m pytest tests/ -x --tb=short -q` to confirm full test suite passes
|
||||
- Verify Helm template renders correctly with 0 signal-engine replicas
|
||||
- _Requirements: all_
|
||||
|
||||
- [x] 10. Checkpoint - Ensure all tests pass
|
||||
- Ensure all tests pass, ask the user if questions arise.
|
||||
|
||||
## Task Dependency Graph
|
||||
|
||||
```json
|
||||
{
|
||||
"waves": [
|
||||
{"tasks": ["1", "2"]},
|
||||
{"tasks": ["3", "4"]},
|
||||
{"tasks": ["5"]},
|
||||
{"tasks": ["6"]},
|
||||
{"tasks": ["7"]},
|
||||
{"tasks": ["8"]},
|
||||
{"tasks": ["9"]},
|
||||
{"tasks": ["10"]}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Tasks 3, 4 are independent config changes (no code deps).
|
||||
Task 5 is independent but task 6 builds on the price fallback concept.
|
||||
Task 7 is the most complex (new function + integration).
|
||||
Tasks 8-10 must run after all fixes are applied.
|
||||
|
||||
## Notes
|
||||
|
||||
- Bug 4 (signal engine) is validated by Helm template rendering, not a unit test
|
||||
- The backfill script (6.2) is a one-time operation, not covered by recurring tests
|
||||
- Preservation tests use Hypothesis with `@settings(max_examples=100)` per project conventions
|
||||
- Test files follow `test_pbt_*` naming convention per project standards
|
||||
@@ -17,6 +17,8 @@ services:
|
||||
replicas: 1
|
||||
dashboard:
|
||||
replicas: 1
|
||||
signalEngine:
|
||||
replicas: 0
|
||||
|
||||
## Beta-specific config overrides
|
||||
## Beta shares the paper DB — DEPLOY_STAGE=beta isolates Redis keys
|
||||
|
||||
@@ -52,3 +52,5 @@ ingress:
|
||||
services:
|
||||
extractor:
|
||||
replicas: 1
|
||||
signalEngine:
|
||||
replicas: 0
|
||||
|
||||
@@ -128,7 +128,7 @@ services:
|
||||
limits: { cpu: 200m, memory: 128Mi }
|
||||
|
||||
signalEngine:
|
||||
replicas: 1
|
||||
replicas: 0
|
||||
pipeline: true
|
||||
image: signal-engine
|
||||
command: "python -m services.signal_engine.main"
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
"""Backfill price_at_prediction for existing NULL prediction snapshots.
|
||||
|
||||
One-time migration script that populates price_at_prediction using the
|
||||
extended fallback chain:
|
||||
1. market_snapshots within 24h of generated_at for the ticker
|
||||
2. positions table (current_price) for the ticker
|
||||
|
||||
Run as: .venv/bin/python scripts/backfill_snapshot_prices.py
|
||||
Dry run: .venv/bin/python scripts/backfill_snapshot_prices.py --dry-run
|
||||
|
||||
Requires env vars: POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST,
|
||||
POSTGRES_PORT, POSTGRES_DB
|
||||
|
||||
Requirements: 2.3
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
import asyncpg # noqa: E402
|
||||
|
||||
from services.shared.config import load_config # noqa: E402
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SQL Queries
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_FIND_NULL_SNAPSHOTS_SQL = """
|
||||
SELECT id, ticker, generated_at
|
||||
FROM prediction_snapshots
|
||||
WHERE price_at_prediction IS NULL
|
||||
ORDER BY generated_at DESC
|
||||
"""
|
||||
|
||||
_MARKET_SNAPSHOT_FALLBACK_SQL = """
|
||||
SELECT (data->>'c')::float AS close
|
||||
FROM market_snapshots
|
||||
WHERE ticker = $1
|
||||
AND snapshot_type = 'bar'
|
||||
AND data->>'c' IS NOT NULL
|
||||
AND captured_at >= $2 - INTERVAL '24 hours'
|
||||
AND captured_at <= $2
|
||||
ORDER BY captured_at DESC
|
||||
LIMIT 1
|
||||
"""
|
||||
|
||||
_POSITIONS_FALLBACK_SQL = """
|
||||
SELECT current_price
|
||||
FROM positions
|
||||
WHERE ticker = $1
|
||||
AND current_price IS NOT NULL
|
||||
LIMIT 1
|
||||
"""
|
||||
|
||||
_UPDATE_PRICE_SQL = """
|
||||
UPDATE prediction_snapshots
|
||||
SET price_at_prediction = $1
|
||||
WHERE id = $2
|
||||
"""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main backfill logic
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def backfill(dry_run: bool = False) -> None:
|
||||
config = load_config()
|
||||
dsn = config.postgres.dsn
|
||||
|
||||
pool = await asyncpg.create_pool(dsn=dsn)
|
||||
assert pool is not None
|
||||
|
||||
# Find all snapshots with NULL price
|
||||
rows = await pool.fetch(_FIND_NULL_SNAPSHOTS_SQL)
|
||||
total = len(rows)
|
||||
|
||||
if total == 0:
|
||||
print("No prediction snapshots with NULL price_at_prediction found.")
|
||||
await pool.close()
|
||||
return
|
||||
|
||||
print(f"Found {total} snapshots with NULL price_at_prediction")
|
||||
if dry_run:
|
||||
print("[DRY RUN] No updates will be performed")
|
||||
print()
|
||||
|
||||
# Statistics
|
||||
found_market = 0
|
||||
found_positions = 0
|
||||
still_null = 0
|
||||
|
||||
for idx, row in enumerate(rows, start=1):
|
||||
snapshot_id = row["id"]
|
||||
ticker = row["ticker"]
|
||||
generated_at = row["generated_at"]
|
||||
|
||||
price: float | None = None
|
||||
|
||||
# Fallback 1: market_snapshots within 24h of generated_at
|
||||
market_row = await pool.fetchrow(
|
||||
_MARKET_SNAPSHOT_FALLBACK_SQL, ticker, generated_at
|
||||
)
|
||||
if market_row and market_row["close"] is not None:
|
||||
price = float(market_row["close"])
|
||||
found_market += 1
|
||||
else:
|
||||
# Fallback 2: positions table
|
||||
pos_row = await pool.fetchrow(_POSITIONS_FALLBACK_SQL, ticker)
|
||||
if pos_row and pos_row["current_price"] is not None:
|
||||
price = float(pos_row["current_price"])
|
||||
found_positions += 1
|
||||
else:
|
||||
still_null += 1
|
||||
|
||||
# Update if we found a price
|
||||
if price is not None and not dry_run:
|
||||
await pool.execute(_UPDATE_PRICE_SQL, price, snapshot_id)
|
||||
|
||||
# Progress reporting every 100 snapshots
|
||||
if idx % 100 == 0:
|
||||
action = "checked" if dry_run else "processed"
|
||||
print(
|
||||
f" {action} {idx}/{total} snapshots "
|
||||
f"(market: {found_market}, positions: {found_positions}, "
|
||||
f"null: {still_null})"
|
||||
)
|
||||
|
||||
await pool.close()
|
||||
|
||||
# Final statistics
|
||||
print()
|
||||
print("=" * 60)
|
||||
print("Backfill complete" if not dry_run else "Dry run complete")
|
||||
print("=" * 60)
|
||||
print(f" Total snapshots processed: {total}")
|
||||
print(f" Found via market_snapshots: {found_market}")
|
||||
print(f" Found via positions: {found_positions}")
|
||||
print(f" Still NULL (no data): {still_null}")
|
||||
if dry_run:
|
||||
updated = found_market + found_positions
|
||||
print(f"\n [DRY RUN] Would have updated {updated} snapshots")
|
||||
print()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Backfill price_at_prediction for NULL prediction snapshots"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Report what would be updated without making changes",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
asyncio.run(backfill(dry_run=args.dry_run))
|
||||
except Exception as e:
|
||||
print(f"Backfill failed: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -15,6 +15,8 @@ from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
import asyncpg
|
||||
|
||||
from services.shared.schemas import MarketContext
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -577,6 +579,57 @@ def sentiment_to_numeric(sentiment: str) -> float:
|
||||
return mapping.get(sentiment.lower(), 0.0)
|
||||
|
||||
|
||||
async def normalize_impact_scores(
|
||||
pool: asyncpg.Pool,
|
||||
ticker: str,
|
||||
raw_scores: list[float],
|
||||
) -> list[float]:
|
||||
"""Normalize impact scores using 7-day rolling z-score per ticker.
|
||||
|
||||
Computes the 7-day mean and stddev of impact_score values from
|
||||
document_impact_records for the given ticker, then normalizes
|
||||
each raw score: normalized = (raw - mean_7d) / max(stddev_7d, 0.1)
|
||||
|
||||
The 0.1 floor prevents division by near-zero stddev for low-activity tickers.
|
||||
|
||||
Fallback: if fewer than 5 records in the 7-day window, returns raw scores
|
||||
unchanged (insufficient data for meaningful normalization).
|
||||
|
||||
Args:
|
||||
pool: asyncpg connection pool.
|
||||
ticker: The ticker symbol to query distribution for.
|
||||
raw_scores: List of raw impact_score values to normalize.
|
||||
|
||||
Returns:
|
||||
List of normalized impact scores (same length as input).
|
||||
"""
|
||||
if not raw_scores:
|
||||
return []
|
||||
|
||||
row = await pool.fetchrow(
|
||||
"""
|
||||
SELECT AVG(impact_score) as mean,
|
||||
STDDEV(impact_score) as stddev,
|
||||
COUNT(*) as cnt
|
||||
FROM document_impact_records
|
||||
WHERE ticker = $1 AND created_at >= NOW() - INTERVAL '7 days'
|
||||
""",
|
||||
ticker,
|
||||
)
|
||||
|
||||
# Fallback: insufficient data for meaningful normalization
|
||||
if row is None or row["cnt"] < 5:
|
||||
return list(raw_scores)
|
||||
|
||||
mean_7d: float = float(row["mean"])
|
||||
stddev_7d: float = float(row["stddev"]) if row["stddev"] is not None and row["stddev"] > 0 else 0.0
|
||||
|
||||
# Apply 0.1 floor to prevent division by near-zero stddev
|
||||
effective_stddev = max(stddev_7d, 0.1)
|
||||
|
||||
return [(raw - mean_7d) / effective_stddev for raw in raw_scores]
|
||||
|
||||
|
||||
def weighted_sentiment_average(signals: list[WeightedSignal]) -> float:
|
||||
"""Compute a weight-adjusted average sentiment across signals.
|
||||
|
||||
|
||||
@@ -51,6 +51,7 @@ from services.aggregation.scoring import (
|
||||
ScoringConfig,
|
||||
WeightedSignal,
|
||||
compute_signal_weight,
|
||||
normalize_impact_scores,
|
||||
sentiment_to_numeric,
|
||||
weighted_sentiment_average,
|
||||
)
|
||||
@@ -2312,6 +2313,13 @@ async def aggregate_company_window(
|
||||
# 2. Fetch market context
|
||||
market_ctx = await fetch_market_context(pool, ticker, window, reference_time)
|
||||
|
||||
# 2a. Normalize impact scores using 7-day z-score per ticker
|
||||
if impacts:
|
||||
raw_scores = [imp.impact_score for imp in impacts]
|
||||
normalized_scores = await normalize_impact_scores(pool, ticker, raw_scores)
|
||||
for imp, norm_score in zip(impacts, normalized_scores):
|
||||
imp.impact_score = norm_score
|
||||
|
||||
# 3. Build weighted signals — pass source accuracy and market data
|
||||
# when in probabilistic mode (Req 4.1–4.3, 6.1–6.5)
|
||||
signals = build_weighted_signals(
|
||||
|
||||
@@ -866,10 +866,10 @@ async def main() -> None:
|
||||
await rds.close()
|
||||
|
||||
|
||||
# How long a document can sit in "parsed" before we consider it orphaned
|
||||
# Must be longer than the expected queue drain time to avoid re-enqueuing
|
||||
# docs that are already queued but not yet processed.
|
||||
STALE_PARSED_THRESHOLD_MINUTES: int = 240
|
||||
# How long a document can sit in "parsed" before we consider it orphaned.
|
||||
# Documents stuck longer than 30 minutes are likely orphaned (Redis lost
|
||||
# their queue entries during pod restart, OOM, etc.).
|
||||
STALE_PARSED_THRESHOLD_MINUTES: int = 30
|
||||
|
||||
# How long after an extraction failure before we retry
|
||||
EXTRACTION_FAILED_RETRY_MINUTES: int = 60
|
||||
@@ -877,7 +877,7 @@ EXTRACTION_FAILED_RETRY_MINUTES: int = 60
|
||||
# Redis set key for tracking enqueued doc IDs (prevents duplicate enqueuing)
|
||||
_ENQUEUED_SET = f"{QUEUE_PREFIX}:enqueued"
|
||||
# How long an enqueued marker lives before it can be re-enqueued (seconds)
|
||||
_ENQUEUED_TTL = 14400 # 4 hours — matches STALE_PARSED_THRESHOLD_MINUTES
|
||||
_ENQUEUED_TTL = 3600 # 1 hour — matches the new recovery cadence
|
||||
|
||||
|
||||
async def _enqueue_if_new(
|
||||
@@ -924,7 +924,7 @@ async def recover_stale_documents(pool: asyncpg.Pool, rds: aioredis.Redis) -> in
|
||||
SELECT 1 FROM global_events ge WHERE ge.source_document_id = d.id
|
||||
)
|
||||
ORDER BY d.created_at ASC
|
||||
LIMIT 100""",
|
||||
LIMIT 500""",
|
||||
STALE_PARSED_THRESHOLD_MINUTES,
|
||||
)
|
||||
|
||||
|
||||
@@ -34,7 +34,7 @@ class QualityGateConfig:
|
||||
min_win_rate: float = 0.53
|
||||
max_ece: float = 0.15
|
||||
min_excess_return_vs_spy: float = 0.0
|
||||
max_snapshot_age_hours: int = 24
|
||||
max_snapshot_age_hours: int = 48
|
||||
|
||||
|
||||
@dataclass
|
||||
|
||||
@@ -323,7 +323,29 @@ async def create_prediction_snapshot(
|
||||
"Used positions fallback price for %s: %s", ticker, ticker_price
|
||||
)
|
||||
else:
|
||||
logger.warning("No market price available for %s at snapshot time", ticker)
|
||||
# Extended fallback: query market_snapshots within 24h time window
|
||||
extended_row = await pool.fetchrow(
|
||||
"""SELECT (data->>'c')::float AS close
|
||||
FROM market_snapshots
|
||||
WHERE ticker = $1
|
||||
AND snapshot_type = 'bar'
|
||||
AND data->>'c' IS NOT NULL
|
||||
AND captured_at >= NOW() - INTERVAL '24 hours'
|
||||
ORDER BY captured_at DESC
|
||||
LIMIT 1""",
|
||||
ticker,
|
||||
)
|
||||
if extended_row:
|
||||
ticker_price = float(extended_row["close"])
|
||||
logger.info(
|
||||
"Used extended 24h market_snapshots fallback price for %s: %s",
|
||||
ticker,
|
||||
ticker_price,
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"No market price available for %s at snapshot time", ticker
|
||||
)
|
||||
|
||||
spy_price = await fetch_latest_close_price(pool, "SPY")
|
||||
if spy_price is None:
|
||||
|
||||
@@ -0,0 +1,538 @@
|
||||
"""Property-based tests for pipeline health bug condition exploration.
|
||||
|
||||
Feature: pipeline-health-fixes
|
||||
|
||||
These tests encode the EXPECTED (fixed) behavior for each bug. They are
|
||||
designed to FAIL on unfixed code, thereby confirming the bugs exist.
|
||||
DO NOT fix these tests or the code when they fail — failure is the goal.
|
||||
|
||||
Bug conditions tested:
|
||||
1. Batch Overflow — recovery capped at 100 docs (should be 500)
|
||||
2. Price Fallback Gap — NULL price when market_snapshots has 24h data
|
||||
3. Sentiment Bias — raw impact_score bias produces skewed sentiment
|
||||
5. Quality Gate Staleness — 26h snapshot rejected (should accept up to 48h)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from hypothesis import given, settings
|
||||
from hypothesis import strategies as st
|
||||
|
||||
from services.trading.model_quality_gate import (
|
||||
QualityGateConfig,
|
||||
evaluate_quality_gate,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bug 1: Batch Overflow — recovery should handle up to 500 docs per cycle
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBug1BatchOverflow:
|
||||
"""Bug 1: Stuck Parsed Docs — Batch Overflow
|
||||
|
||||
When 600 documents are stuck in 'parsed' status older than 30 minutes,
|
||||
recover_stale_documents() should process up to 500 per cycle.
|
||||
|
||||
The unfixed code has LIMIT 100 and STALE_PARSED_THRESHOLD_MINUTES=240,
|
||||
so it will only recover at most 100 docs (and won't recover docs only
|
||||
30 min old since threshold is 240 min).
|
||||
|
||||
**Validates: Requirements 2.1**
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recovery_processes_more_than_100_documents(self):
|
||||
"""Create 600 stale docs, assert recovery processes > 100.
|
||||
|
||||
The unfixed code caps at LIMIT 100 in the SQL, so pool.fetch will
|
||||
return at most 100 rows. We verify the function's SQL uses a limit
|
||||
that allows processing more than 100 documents.
|
||||
"""
|
||||
from services.scheduler.app import (
|
||||
STALE_PARSED_THRESHOLD_MINUTES,
|
||||
recover_stale_documents,
|
||||
)
|
||||
|
||||
# Generate 600 fake document rows (all older than threshold)
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
stale_time = now - timedelta(minutes=STALE_PARSED_THRESHOLD_MINUTES + 10)
|
||||
|
||||
fake_rows = []
|
||||
for i in range(600):
|
||||
row = {
|
||||
"id": uuid.uuid4(),
|
||||
"document_type": "news",
|
||||
"ticker": "AAPL",
|
||||
"updated_at": stale_time,
|
||||
}
|
||||
fake_rows.append(row)
|
||||
|
||||
# Mock pool and redis
|
||||
pool = AsyncMock()
|
||||
# The SQL query in the function has LIMIT 100, so even if we want 600,
|
||||
# pool.fetch will be called with the SQL that has LIMIT 100.
|
||||
# We simulate: the DB returns up to what the LIMIT allows.
|
||||
# On unfixed code: LIMIT 100 → max 100 rows returned.
|
||||
# On fixed code: LIMIT 500 → up to 500 rows returned.
|
||||
# We'll return all 600 (simulating DB has 600 matching rows) and let
|
||||
# the SQL LIMIT be the constraint. Since we're mocking, we return
|
||||
# based on what the actual limit would be.
|
||||
# Actually, since pool.fetch is mocked, we need to check the SQL.
|
||||
# The simplest approach: return 500 rows and check how many get processed.
|
||||
# If the code has LIMIT 100, it will only process what pool.fetch returns.
|
||||
# But since pool.fetch is mocked, we return 500 to test the upper bound.
|
||||
pool.fetch = AsyncMock(return_value=fake_rows[:500])
|
||||
pool.execute = AsyncMock()
|
||||
|
||||
rds = AsyncMock()
|
||||
# Every _enqueue_if_new call succeeds (marker key doesn't exist)
|
||||
rds.set = AsyncMock(return_value=True)
|
||||
rds.rpush = AsyncMock()
|
||||
|
||||
result = await recover_stale_documents(pool, rds)
|
||||
|
||||
# The fixed code should process up to 500 documents.
|
||||
# The unfixed code has LIMIT 100, so pool.fetch returns at most 100.
|
||||
# But since we mock pool.fetch to return 500, the real constraint is
|
||||
# the STALE_PARSED_THRESHOLD_MINUTES. On unfixed code (240 min threshold),
|
||||
# docs that are only 30 min old won't be recovered.
|
||||
# Let's verify via the threshold instead:
|
||||
# The key assertion: the threshold should be 30 min (not 240)
|
||||
# so that 30-min-old documents ARE recovered.
|
||||
assert STALE_PARSED_THRESHOLD_MINUTES <= 30, (
|
||||
f"Bug 1 confirmed: STALE_PARSED_THRESHOLD_MINUTES is "
|
||||
f"{STALE_PARSED_THRESHOLD_MINUTES} (should be <= 30)"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sql_batch_limit_is_500(self):
|
||||
"""Verify the SQL query uses LIMIT 500 (not LIMIT 100).
|
||||
|
||||
Inspects the actual SQL passed to pool.fetch to confirm the batch
|
||||
limit has been increased.
|
||||
"""
|
||||
from services.scheduler.app import recover_stale_documents
|
||||
|
||||
pool = AsyncMock()
|
||||
pool.fetch = AsyncMock(return_value=[])
|
||||
pool.execute = AsyncMock()
|
||||
|
||||
rds = AsyncMock()
|
||||
|
||||
await recover_stale_documents(pool, rds)
|
||||
|
||||
# Check the SQL query that was passed to pool.fetch
|
||||
call_args = pool.fetch.call_args
|
||||
sql_query = call_args[0][0]
|
||||
|
||||
assert "LIMIT 500" in sql_query, (
|
||||
f"Bug 1 confirmed: SQL uses '{sql_query.split('LIMIT')[1].strip()[:10]}...' "
|
||||
f"(expected LIMIT 500)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bug 2: Price Fallback Gap — missing 24h market_snapshots fallback
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBug2PriceFallbackGap:
|
||||
"""Bug 2: Extended Price Fallback
|
||||
|
||||
When creating a prediction snapshot for a ticker with:
|
||||
- No exact market_snapshots match (fetch_latest_close_price → None)
|
||||
- No position in positions table
|
||||
- But market_snapshots has data within 24h
|
||||
|
||||
The price_at_prediction should NOT be NULL. The unfixed code only falls
|
||||
back to positions and doesn't have the 24h time-window query.
|
||||
|
||||
**Validates: Requirements 2.2, 2.3**
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_price_not_null_when_24h_data_exists(self):
|
||||
"""Assert that the code has a 24h time-window fallback for price lookup.
|
||||
|
||||
The unfixed code in create_prediction_snapshot() only falls back to
|
||||
the positions table after fetch_latest_close_price returns None.
|
||||
There is NO query for market_snapshots within a 24h time window.
|
||||
|
||||
We verify the fallback chain by inspecting what happens when:
|
||||
1. fetch_latest_close_price → None (no exact match)
|
||||
2. positions table → None (no open position)
|
||||
3. 24h market_snapshots query → returns a price (FIXED code only)
|
||||
"""
|
||||
from services.validation.prediction_snapshot import (
|
||||
create_prediction_snapshot,
|
||||
)
|
||||
|
||||
# Mock pool with full transaction context manager support
|
||||
pool = AsyncMock()
|
||||
|
||||
# Simulate the fallback chain:
|
||||
# - 1st fetchrow: positions table → None
|
||||
# - 2nd fetchrow: 24h market_snapshots → returns price 42.50
|
||||
# - 3rd fetchrow: SPY via fetch_latest_close_price (patched separately)
|
||||
# - 4th fetchrow: sector lookup → None
|
||||
pool.fetchrow = AsyncMock(
|
||||
side_effect=[
|
||||
None, # positions table lookup
|
||||
{"close": 42.50}, # 24h market_snapshots extended fallback
|
||||
None, # sector ETF lookup (_fetch_sector_etf_ticker)
|
||||
]
|
||||
)
|
||||
pool.execute = AsyncMock()
|
||||
|
||||
# Setup transaction context manager mock
|
||||
conn_mock = AsyncMock()
|
||||
conn_mock.execute = AsyncMock()
|
||||
tx_mock = AsyncMock()
|
||||
tx_mock.__aenter__ = AsyncMock(return_value=None)
|
||||
tx_mock.__aexit__ = AsyncMock(return_value=None)
|
||||
conn_mock.transaction = MagicMock(return_value=tx_mock)
|
||||
acquire_mock = AsyncMock()
|
||||
acquire_mock.__aenter__ = AsyncMock(return_value=conn_mock)
|
||||
acquire_mock.__aexit__ = AsyncMock(return_value=None)
|
||||
pool.acquire = MagicMock(return_value=acquire_mock)
|
||||
|
||||
# Build minimal mocks for Recommendation and TrendSummary
|
||||
recommendation = MagicMock()
|
||||
recommendation.ticker = "PLTR"
|
||||
recommendation.generated_at = datetime.now(tz=timezone.utc)
|
||||
recommendation.time_horizon = "7d"
|
||||
recommendation.action.value = "buy"
|
||||
recommendation.mode.value = "paper_eligible"
|
||||
recommendation.confidence = 0.6
|
||||
|
||||
trend_summary = MagicMock()
|
||||
trend_summary.market_context = None
|
||||
trend_summary.window.value = "7d"
|
||||
trend_summary.trend_direction.value = "bullish"
|
||||
trend_summary.trend_strength = 0.7
|
||||
trend_summary.contradiction_score = 0.1
|
||||
trend_summary.p_bull = 0.7
|
||||
|
||||
# Patch fetch_latest_close_price to return None (no exact match)
|
||||
with patch(
|
||||
"services.validation.prediction_snapshot.fetch_latest_close_price",
|
||||
new_callable=AsyncMock,
|
||||
return_value=None,
|
||||
):
|
||||
snapshot = await create_prediction_snapshot(
|
||||
pool=pool,
|
||||
recommendation=recommendation,
|
||||
trend_summary=trend_summary,
|
||||
evidence_signals=[],
|
||||
evidence_docs=[],
|
||||
)
|
||||
|
||||
# On FIXED code: price_at_prediction should NOT be None because
|
||||
# the 24h market_snapshots fallback finds data (42.50).
|
||||
# On UNFIXED code: price_at_prediction IS None (bug confirmed).
|
||||
assert snapshot.price_at_prediction is not None, (
|
||||
"Bug 2 confirmed: price_at_prediction is NULL when no position exists. "
|
||||
"The code lacks a 24h time-window market_snapshots fallback query."
|
||||
)
|
||||
|
||||
def test_fallback_chain_has_24h_market_query(self):
|
||||
"""Verify create_prediction_snapshot source code contains a 24h fallback.
|
||||
|
||||
The unfixed code only has two price sources:
|
||||
1. fetch_latest_close_price() — exact timestamp match
|
||||
2. positions table — current_price for held tickers
|
||||
|
||||
The FIXED code should add a third:
|
||||
3. market_snapshots within 24 hours — time-window query
|
||||
|
||||
We inspect the source to verify the 24h fallback exists.
|
||||
"""
|
||||
import inspect
|
||||
|
||||
from services.validation.prediction_snapshot import (
|
||||
create_prediction_snapshot,
|
||||
)
|
||||
|
||||
source = inspect.getsource(create_prediction_snapshot)
|
||||
|
||||
# Look for evidence of a 24-hour time-window fallback query
|
||||
has_24h_fallback = (
|
||||
"24 hours" in source
|
||||
or "24h" in source
|
||||
or "INTERVAL" in source and "24" in source
|
||||
or "timedelta(hours=24)" in source
|
||||
)
|
||||
|
||||
assert has_24h_fallback, (
|
||||
"Bug 2 confirmed: create_prediction_snapshot() does not contain a "
|
||||
"24h time-window fallback query for market_snapshots. "
|
||||
"When fetch_latest_close_price returns None and positions table "
|
||||
"has no data, price_at_prediction will be NULL."
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bug 3: Sentiment Bias — raw impact_score bias not normalized
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBug3SentimentBias:
|
||||
"""Bug 3: Sentiment Z-Score Normalization
|
||||
|
||||
When impact_score values have systematic negative bias (e.g., NuExtract3
|
||||
outputs clustered around -0.3), the weighted_sentiment_average() produces
|
||||
biased negative signals. The fix adds normalize_impact_scores() which
|
||||
zero-centers scores BEFORE they reach weighted_sentiment_average().
|
||||
|
||||
weighted_sentiment_average() is a pure function — normalization happens
|
||||
upstream in build_weighted_signals() via normalize_impact_scores().
|
||||
|
||||
**Validates: Requirements 2.4**
|
||||
"""
|
||||
|
||||
@given(
|
||||
# Generate 50 impact scores with systematic negative bias
|
||||
bias_mean=st.floats(min_value=-0.5, max_value=-0.1, allow_nan=False),
|
||||
bias_stddev=st.floats(min_value=0.05, max_value=0.2, allow_nan=False),
|
||||
)
|
||||
@settings(max_examples=100)
|
||||
def test_normalize_impact_scores_zero_centers_biased_input(
|
||||
self,
|
||||
bias_mean: float,
|
||||
bias_stddev: float,
|
||||
):
|
||||
"""Verify normalize_impact_scores() produces zero-centered output.
|
||||
|
||||
The fix adds normalize_impact_scores() which applies z-score
|
||||
normalization: (raw - mean_7d) / max(stddev_7d, 0.1).
|
||||
When given biased raw scores and sufficient 7-day history with
|
||||
matching stats, the output should be approximately zero-centered.
|
||||
|
||||
This tests the normalization function directly (the fix for Bug 3),
|
||||
since weighted_sentiment_average() is a pure function that receives
|
||||
already-normalized scores from the upstream pipeline.
|
||||
"""
|
||||
import random
|
||||
|
||||
from services.aggregation.scoring import normalize_impact_scores
|
||||
|
||||
random.seed(42)
|
||||
|
||||
# Generate 50 biased raw impact scores
|
||||
raw_scores = [
|
||||
max(-1.0, min(1.0, random.gauss(bias_mean, bias_stddev)))
|
||||
for _ in range(50)
|
||||
]
|
||||
|
||||
# Compute actual mean and stddev of the raw scores (simulating 7-day stats)
|
||||
actual_mean = sum(raw_scores) / len(raw_scores)
|
||||
actual_stddev = (
|
||||
sum((x - actual_mean) ** 2 for x in raw_scores) / len(raw_scores)
|
||||
) ** 0.5
|
||||
|
||||
# Mock pool to return the biased distribution's statistics
|
||||
# (simulating that the 7-day history reflects the same bias)
|
||||
mock_pool = AsyncMock()
|
||||
mock_pool.fetchrow = AsyncMock(
|
||||
return_value={
|
||||
"mean": actual_mean,
|
||||
"stddev": actual_stddev,
|
||||
"cnt": 50,
|
||||
}
|
||||
)
|
||||
|
||||
# Run normalization using asyncio.run() for Hypothesis compatibility
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
normalized = loop.run_until_complete(
|
||||
normalize_impact_scores(mock_pool, "AAPL", raw_scores)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# After z-score normalization, mean should be ≈ 0
|
||||
normalized_mean = sum(normalized) / len(normalized)
|
||||
assert abs(normalized_mean) < 0.15, (
|
||||
f"Bug 3 confirmed: normalize_impact_scores does not zero-center "
|
||||
f"biased input. Output mean = {normalized_mean:.4f} "
|
||||
f"(expected ≈ 0, input bias_mean={bias_mean:.3f})"
|
||||
)
|
||||
|
||||
def test_normalize_impact_scores_exists_and_is_integrated(self):
|
||||
"""Verify normalize_impact_scores is available and integrated in the worker.
|
||||
|
||||
The unfixed code does not have this function. Its existence and use
|
||||
in the aggregation worker confirms the fix has been applied.
|
||||
"""
|
||||
import inspect
|
||||
|
||||
from services.aggregation import worker as worker_module
|
||||
from services.aggregation.scoring import normalize_impact_scores
|
||||
|
||||
# Verify normalize_impact_scores exists as an async function
|
||||
assert inspect.iscoroutinefunction(normalize_impact_scores), (
|
||||
"Bug 3 confirmed: normalize_impact_scores is not an async function"
|
||||
)
|
||||
|
||||
# Verify it's imported and used in the worker module
|
||||
worker_source = inspect.getsource(worker_module)
|
||||
assert "normalize_impact_scores" in worker_source, (
|
||||
"Bug 3 confirmed: aggregation worker does not call "
|
||||
"normalize_impact_scores — raw impact_scores flow through unmodified"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_normalization_with_alternating_sentiment_produces_centered_output(
|
||||
self,
|
||||
):
|
||||
"""After normalization, the mean of normalized scores is ≈ 0.
|
||||
|
||||
This verifies the full normalization path: given biased raw scores
|
||||
and matching 7-day statistics, normalize_impact_scores produces
|
||||
zero-centered output suitable for unbiased downstream weighting.
|
||||
"""
|
||||
import random
|
||||
|
||||
from services.aggregation.scoring import normalize_impact_scores
|
||||
|
||||
random.seed(42)
|
||||
bias_mean = -0.3
|
||||
bias_stddev = 0.1
|
||||
|
||||
# Generate 50 biased raw scores
|
||||
raw_scores = [
|
||||
max(-1.0, min(1.0, random.gauss(bias_mean, bias_stddev)))
|
||||
for _ in range(50)
|
||||
]
|
||||
|
||||
actual_mean = sum(raw_scores) / len(raw_scores)
|
||||
actual_stddev = (
|
||||
sum((x - actual_mean) ** 2 for x in raw_scores) / len(raw_scores)
|
||||
) ** 0.5
|
||||
|
||||
# Mock pool with 7-day stats matching the biased distribution
|
||||
mock_pool = AsyncMock()
|
||||
mock_pool.fetchrow = AsyncMock(
|
||||
return_value={
|
||||
"mean": actual_mean,
|
||||
"stddev": actual_stddev,
|
||||
"cnt": 50,
|
||||
}
|
||||
)
|
||||
|
||||
# Normalize scores
|
||||
normalized = await normalize_impact_scores(mock_pool, "AAPL", raw_scores)
|
||||
|
||||
# After z-score normalization, the mean of normalized scores should be ≈ 0
|
||||
normalized_mean = sum(normalized) / len(normalized)
|
||||
assert abs(normalized_mean) < 0.01, (
|
||||
f"Bug 3 confirmed: normalize_impact_scores output is not zero-centered. "
|
||||
f"Mean = {normalized_mean:.6f} (expected ≈ 0)"
|
||||
)
|
||||
|
||||
# And the original biased mean should have been removed
|
||||
raw_mean = sum(raw_scores) / len(raw_scores)
|
||||
assert abs(raw_mean) > 0.1, (
|
||||
"Test setup issue: raw scores should have significant bias"
|
||||
)
|
||||
assert abs(normalized_mean) < abs(raw_mean), (
|
||||
f"Bug 3 confirmed: normalization did not reduce bias. "
|
||||
f"Raw mean={raw_mean:.4f}, normalized mean={normalized_mean:.4f}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bug 5: Quality Gate Staleness — 24h threshold too strict
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBug5QualityGateStaleness:
|
||||
"""Bug 5: Quality Gate Threshold
|
||||
|
||||
When the most recent metric snapshot is 26 hours old (between 24h and 48h),
|
||||
the quality gate should PASS (accept it). The unfixed code has
|
||||
max_snapshot_age_hours=24 which rejects it as stale.
|
||||
|
||||
**Validates: Requirements 2.6**
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_26h_old_snapshot_passes_quality_gate(self):
|
||||
"""A 26h-old snapshot should pass the quality gate.
|
||||
|
||||
The unfixed code has max_snapshot_age_hours=24, so a 26h-old snapshot
|
||||
is rejected as stale and forces paper-only mode.
|
||||
"""
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
snapshot_time = now - timedelta(hours=26)
|
||||
|
||||
# Build a valid snapshot row that meets all metric thresholds
|
||||
fake_snapshot_row = {
|
||||
"id": uuid.uuid4(),
|
||||
"generated_at": snapshot_time,
|
||||
"prediction_count": 200,
|
||||
"win_rate": 0.60,
|
||||
"directional_accuracy": 0.58,
|
||||
"information_coefficient": 0.05,
|
||||
"rank_information_coefficient": 0.04,
|
||||
"avg_return": 0.02,
|
||||
"avg_excess_return_vs_spy": 0.01,
|
||||
"avg_excess_return_vs_sector": 0.005,
|
||||
"calibration_error": 0.10,
|
||||
"brier_score": 0.20,
|
||||
"buy_win_rate": 0.62,
|
||||
"sell_win_rate": 0.58,
|
||||
"hold_win_rate": 0.55,
|
||||
}
|
||||
|
||||
pool = AsyncMock()
|
||||
pool.fetchrow = AsyncMock(return_value=fake_snapshot_row)
|
||||
pool.execute = AsyncMock()
|
||||
pool.fetchval = AsyncMock(return_value=None)
|
||||
|
||||
# Use default config (which has max_snapshot_age_hours=24 on unfixed code)
|
||||
config = QualityGateConfig()
|
||||
|
||||
# Patch _store_gate_result to avoid DB writes
|
||||
with patch(
|
||||
"services.trading.model_quality_gate._store_gate_result",
|
||||
new_callable=AsyncMock,
|
||||
):
|
||||
# Also patch load_gate_config_from_db since we provide config directly
|
||||
result = await evaluate_quality_gate(pool, config=config)
|
||||
|
||||
# On FIXED code (max_snapshot_age_hours=48): result.passed should be True
|
||||
# because 26h < 48h and all metrics meet thresholds.
|
||||
# On UNFIXED code (max_snapshot_age_hours=24): result.passed is False
|
||||
# because 26h > 24h triggers the staleness check.
|
||||
assert result.passed is True, (
|
||||
f"Bug 5 confirmed: quality gate rejects 26h-old snapshot as stale. "
|
||||
f"Reason: '{result.reason}'. "
|
||||
f"max_snapshot_age_hours={config.max_snapshot_age_hours} "
|
||||
f"(should be 48)"
|
||||
)
|
||||
|
||||
@given(
|
||||
age_hours=st.floats(min_value=24.1, max_value=47.9, allow_nan=False),
|
||||
)
|
||||
@settings(max_examples=50)
|
||||
def test_snapshots_between_24h_and_48h_should_pass(self, age_hours: float):
|
||||
"""Property: any snapshot aged [24h, 48h) with good metrics should pass.
|
||||
|
||||
The unfixed code rejects all snapshots > 24h.
|
||||
"""
|
||||
config = QualityGateConfig()
|
||||
|
||||
# The default config has max_snapshot_age_hours = 24 on unfixed code
|
||||
# The fixed code should have max_snapshot_age_hours = 48
|
||||
assert config.max_snapshot_age_hours >= 48, (
|
||||
f"Bug 5 confirmed: QualityGateConfig.max_snapshot_age_hours = "
|
||||
f"{config.max_snapshot_age_hours} (should be >= 48 to accept "
|
||||
f"snapshots up to 48h old, but got {age_hours:.1f}h snapshot rejected)"
|
||||
)
|
||||
@@ -0,0 +1,466 @@
|
||||
"""Property-based tests for pipeline health preservation properties.
|
||||
|
||||
Feature: pipeline-health-fixes
|
||||
|
||||
These tests encode the CURRENT correct behavior for non-bug inputs.
|
||||
They MUST PASS on unfixed code — confirming baseline behavior is preserved.
|
||||
|
||||
Preservation properties tested:
|
||||
1. Normal Doc Processing — documents < threshold age untouched by recovery
|
||||
2. Primary Price Path — valid primary price used without fallback
|
||||
3. Balanced Sentiment — symmetric impact_scores produce neutral output
|
||||
4. Quality Gate Normal — snapshots < max age meeting criteria pass
|
||||
5. Failed Extraction Independence — retry_failed_extractions behavior unchanged
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from hypothesis import given, settings
|
||||
from hypothesis import strategies as st
|
||||
|
||||
from services.aggregation.scoring import (
|
||||
SignalWeight,
|
||||
WeightedSignal,
|
||||
weighted_sentiment_average,
|
||||
)
|
||||
from services.trading.model_quality_gate import (
|
||||
QualityGateConfig,
|
||||
evaluate_quality_gate,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Property 1: Normal Doc Processing Preservation
|
||||
# Documents younger than the stale threshold are NOT touched by recovery.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPreservationNormalDocProcessing:
|
||||
"""Preservation: Normal Document Processing
|
||||
|
||||
Documents that enter 'parsed' status and are younger than the current
|
||||
stale threshold are never touched by recover_stale_documents().
|
||||
On unfixed code, STALE_PARSED_THRESHOLD_MINUTES=240, so any document
|
||||
younger than 240 min is untouched.
|
||||
|
||||
We test with documents < 30 min old which are well within the threshold
|
||||
on both unfixed (240 min) and fixed (30 min) code.
|
||||
|
||||
**Validates: Requirements 3.1, 3.7**
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@given(
|
||||
doc_age_minutes=st.floats(min_value=0.0, max_value=29.0, allow_nan=False),
|
||||
)
|
||||
@settings(max_examples=100)
|
||||
async def test_young_documents_not_recovered(self, doc_age_minutes: float):
|
||||
"""Property: for all documents with age < 30 min, recovery does NOT enqueue them.
|
||||
|
||||
The SQL WHERE clause filters on `updated_at < NOW() - INTERVAL threshold`,
|
||||
so documents younger than the threshold won't appear in the query results.
|
||||
We mock pool.fetch to return empty (simulating the DB correctly filtering
|
||||
out young documents) and verify 0 are enqueued.
|
||||
"""
|
||||
from services.scheduler.app import (
|
||||
STALE_PARSED_THRESHOLD_MINUTES,
|
||||
recover_stale_documents,
|
||||
)
|
||||
|
||||
# The current threshold is 240 min on unfixed code.
|
||||
# Documents < 30 min old are well below ANY threshold (240 or 30),
|
||||
# so the DB query returns nothing for them.
|
||||
assert doc_age_minutes < STALE_PARSED_THRESHOLD_MINUTES, (
|
||||
f"Test assumes doc_age_minutes ({doc_age_minutes}) < threshold "
|
||||
f"({STALE_PARSED_THRESHOLD_MINUTES})"
|
||||
)
|
||||
|
||||
# Mock pool.fetch to return empty — simulating DB filtering out young docs
|
||||
pool = AsyncMock()
|
||||
pool.fetch = AsyncMock(return_value=[])
|
||||
pool.execute = AsyncMock()
|
||||
|
||||
rds = AsyncMock()
|
||||
rds.set = AsyncMock(return_value=True)
|
||||
rds.rpush = AsyncMock()
|
||||
|
||||
result = await recover_stale_documents(pool, rds)
|
||||
|
||||
# No documents should be enqueued
|
||||
assert result == 0, (
|
||||
f"Young documents (age={doc_age_minutes:.1f} min) should not be "
|
||||
f"recovered, but {result} were enqueued"
|
||||
)
|
||||
# Redis rpush should never have been called
|
||||
rds.rpush.assert_not_called()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Property 2: Primary Price Path Preservation
|
||||
# When fetch_latest_close_price returns a valid price, no fallback is invoked.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPreservationPrimaryPricePath:
|
||||
"""Preservation: Primary Price Path
|
||||
|
||||
When fetch_latest_close_price() returns a valid price, that price is used
|
||||
directly and no fallback (positions table or market_snapshots 24h) is
|
||||
invoked.
|
||||
|
||||
**Validates: Requirements 3.2**
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@given(
|
||||
primary_price=st.floats(
|
||||
min_value=1.0, max_value=5000.0, allow_nan=False, allow_infinity=False
|
||||
),
|
||||
)
|
||||
@settings(max_examples=100)
|
||||
async def test_valid_primary_price_used_directly(self, primary_price: float):
|
||||
"""Property: for all tickers where primary price exists, result equals primary price.
|
||||
|
||||
When fetch_latest_close_price returns a valid float, the snapshot's
|
||||
price_at_prediction equals that value without any fallback invoked.
|
||||
"""
|
||||
from services.validation.prediction_snapshot import (
|
||||
create_prediction_snapshot,
|
||||
)
|
||||
|
||||
pool = AsyncMock()
|
||||
|
||||
# Track fetchrow calls to verify no fallback query is made
|
||||
call_count = {"fetchrow": 0}
|
||||
|
||||
async def mock_fetchrow(*args, **kwargs):
|
||||
call_count["fetchrow"] += 1
|
||||
# After the first price lookups, return None for sector ETF
|
||||
return None
|
||||
|
||||
pool.fetchrow = AsyncMock(side_effect=mock_fetchrow)
|
||||
pool.execute = AsyncMock()
|
||||
|
||||
# Setup transaction context manager mock
|
||||
conn_mock = AsyncMock()
|
||||
conn_mock.execute = AsyncMock()
|
||||
tx_mock = AsyncMock()
|
||||
tx_mock.__aenter__ = AsyncMock(return_value=None)
|
||||
tx_mock.__aexit__ = AsyncMock(return_value=None)
|
||||
conn_mock.transaction = MagicMock(return_value=tx_mock)
|
||||
acquire_mock = AsyncMock()
|
||||
acquire_mock.__aenter__ = AsyncMock(return_value=conn_mock)
|
||||
acquire_mock.__aexit__ = AsyncMock(return_value=None)
|
||||
pool.acquire = MagicMock(return_value=acquire_mock)
|
||||
|
||||
# Build minimal mocks
|
||||
recommendation = MagicMock()
|
||||
recommendation.ticker = "AAPL"
|
||||
recommendation.generated_at = datetime.now(tz=timezone.utc)
|
||||
recommendation.time_horizon = "7d"
|
||||
recommendation.action.value = "buy"
|
||||
recommendation.mode.value = "paper_eligible"
|
||||
recommendation.confidence = 0.7
|
||||
|
||||
trend_summary = MagicMock()
|
||||
trend_summary.market_context = None
|
||||
trend_summary.window.value = "7d"
|
||||
trend_summary.trend_direction.value = "bullish"
|
||||
trend_summary.trend_strength = 0.7
|
||||
trend_summary.contradiction_score = 0.1
|
||||
trend_summary.p_bull = 0.7
|
||||
|
||||
# Patch fetch_latest_close_price to return the primary price
|
||||
# This simulates the primary price lookup succeeding
|
||||
async def mock_fetch_latest_close_price(pool_arg, ticker):
|
||||
if ticker == "AAPL":
|
||||
return primary_price
|
||||
# SPY and sector ETF can also have prices
|
||||
return primary_price * 0.5
|
||||
|
||||
with patch(
|
||||
"services.validation.prediction_snapshot.fetch_latest_close_price",
|
||||
new_callable=AsyncMock,
|
||||
side_effect=mock_fetch_latest_close_price,
|
||||
):
|
||||
snapshot = await create_prediction_snapshot(
|
||||
pool=pool,
|
||||
recommendation=recommendation,
|
||||
trend_summary=trend_summary,
|
||||
evidence_signals=[],
|
||||
evidence_docs=[],
|
||||
)
|
||||
|
||||
# The snapshot price should equal the primary price exactly
|
||||
assert snapshot.price_at_prediction == primary_price, (
|
||||
f"Primary price path broken: expected {primary_price}, "
|
||||
f"got {snapshot.price_at_prediction}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Property 3: Balanced Sentiment Preservation
|
||||
# Tickers with symmetric impact_score distributions produce neutral signals.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPreservationBalancedSentiment:
|
||||
"""Preservation: Balanced Sentiment
|
||||
|
||||
Tickers with symmetric impact_score distributions (mean ≈ 0) produce
|
||||
neutral signals. The weighted_sentiment_average() with equal positive
|
||||
and negative sentiments and impact_scores centered around 0 should
|
||||
produce an output near 0.
|
||||
|
||||
**Validates: Requirements 3.3**
|
||||
"""
|
||||
|
||||
@given(
|
||||
# Generate symmetric impact_scores centered at 0
|
||||
stddev=st.floats(min_value=0.1, max_value=1.0, allow_nan=False),
|
||||
n_pairs=st.integers(min_value=5, max_value=50),
|
||||
)
|
||||
@settings(max_examples=100)
|
||||
def test_symmetric_impacts_produce_neutral_output(
|
||||
self,
|
||||
stddev: float,
|
||||
n_pairs: int,
|
||||
):
|
||||
"""Property: for all impact_score sets with mean ≈ 0, output is near 0.
|
||||
|
||||
Generate WeightedSignals with impact_scores drawn symmetrically around 0
|
||||
(for each +x there is a -x) and equal positive/negative sentiments.
|
||||
Verify weighted_sentiment_average() produces near-zero output.
|
||||
"""
|
||||
signals: list[WeightedSignal] = []
|
||||
|
||||
for i in range(n_pairs):
|
||||
# Create symmetric pairs of impact_scores
|
||||
# Use positive impact_score values (the scoring formula uses w = combined * impact_score)
|
||||
# With symmetric sentiments and equal positive impact_scores, output should be 0
|
||||
impact_val = 0.1 + (i * stddev / n_pairs) # All positive, equal for both
|
||||
|
||||
weight = SignalWeight(
|
||||
recency=0.8,
|
||||
credibility=0.7,
|
||||
novelty_bonus=0.1,
|
||||
confidence_gate=1.0,
|
||||
market_ctx_multiplier=1.0,
|
||||
combined=0.5,
|
||||
)
|
||||
|
||||
# Positive sentiment signal
|
||||
signals.append(
|
||||
WeightedSignal(
|
||||
document_id=f"doc_pos_{i}",
|
||||
weight=weight,
|
||||
sentiment_value=1.0,
|
||||
impact_score=impact_val,
|
||||
)
|
||||
)
|
||||
# Negative sentiment signal with same impact_score
|
||||
signals.append(
|
||||
WeightedSignal(
|
||||
document_id=f"doc_neg_{i}",
|
||||
weight=weight,
|
||||
sentiment_value=-1.0,
|
||||
impact_score=impact_val,
|
||||
)
|
||||
)
|
||||
|
||||
avg = weighted_sentiment_average(signals)
|
||||
|
||||
# With perfectly symmetric positive/negative sentiments and equal
|
||||
# impact_scores, the output should be exactly 0
|
||||
assert abs(avg) < 1e-9, (
|
||||
f"Balanced sentiment broken: expected ≈ 0, got {avg:.6f} "
|
||||
f"with {n_pairs} pairs and stddev={stddev:.3f}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Property 4: Quality Gate Normal Preservation
|
||||
# Snapshots < max_snapshot_age_hours meeting thresholds pass the quality gate.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPreservationQualityGateNormal:
|
||||
"""Preservation: Quality Gate Normal
|
||||
|
||||
Snapshots younger than the max age threshold that meet all metric criteria
|
||||
continue to pass the quality gate and return passed=True.
|
||||
|
||||
On unfixed code, max_snapshot_age_hours=24, so we test with ages in [0, 24h).
|
||||
This ensures the test passes on both unfixed and fixed code.
|
||||
|
||||
**Validates: Requirements 3.5, 3.6**
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@given(
|
||||
age_hours=st.floats(min_value=0.1, max_value=23.9, allow_nan=False),
|
||||
win_rate=st.floats(min_value=0.53, max_value=0.85, allow_nan=False),
|
||||
ic=st.floats(min_value=0.03, max_value=0.3, allow_nan=False),
|
||||
prediction_count=st.integers(min_value=100, max_value=10000),
|
||||
)
|
||||
@settings(max_examples=100)
|
||||
async def test_young_snapshots_meeting_criteria_pass(
|
||||
self,
|
||||
age_hours: float,
|
||||
win_rate: float,
|
||||
ic: float,
|
||||
prediction_count: int,
|
||||
):
|
||||
"""Property: for all snapshot ages in [0, 24h) meeting metric criteria,
|
||||
quality gate returns passed=True.
|
||||
|
||||
We use ages < 24h which is within the threshold on BOTH unfixed (24h)
|
||||
and fixed (48h) code.
|
||||
"""
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
snapshot_time = now - timedelta(hours=age_hours)
|
||||
|
||||
# Build a snapshot row that meets all default thresholds
|
||||
fake_snapshot_row = {
|
||||
"id": uuid.uuid4(),
|
||||
"generated_at": snapshot_time,
|
||||
"prediction_count": prediction_count,
|
||||
"win_rate": win_rate,
|
||||
"directional_accuracy": 0.58,
|
||||
"information_coefficient": ic,
|
||||
"rank_information_coefficient": 0.04,
|
||||
"avg_return": 0.02,
|
||||
"avg_excess_return_vs_spy": 0.01, # >= 0.0 threshold
|
||||
"avg_excess_return_vs_sector": 0.005,
|
||||
"calibration_error": 0.10, # <= 0.15 threshold
|
||||
"brier_score": 0.20,
|
||||
"buy_win_rate": 0.62,
|
||||
"sell_win_rate": 0.58,
|
||||
"hold_win_rate": 0.55,
|
||||
}
|
||||
|
||||
pool = AsyncMock()
|
||||
pool.fetchrow = AsyncMock(return_value=fake_snapshot_row)
|
||||
pool.execute = AsyncMock()
|
||||
pool.fetchval = AsyncMock(return_value=None)
|
||||
|
||||
config = QualityGateConfig()
|
||||
|
||||
with patch(
|
||||
"services.trading.model_quality_gate._store_gate_result",
|
||||
new_callable=AsyncMock,
|
||||
):
|
||||
result = await evaluate_quality_gate(pool, config=config)
|
||||
|
||||
assert result.passed is True, (
|
||||
f"Quality gate should PASS for {age_hours:.1f}h-old snapshot "
|
||||
f"meeting all criteria, but got: {result.reason}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Property 5: Failed Extraction Independence Preservation
|
||||
# retry_failed_extractions() behavior is unaffected by stale doc recovery changes.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPreservationFailedExtractionIndependence:
|
||||
"""Preservation: Failed Extraction Independence
|
||||
|
||||
retry_failed_extractions() processes documents in 'extraction_failed' status
|
||||
using its own threshold (EXTRACTION_FAILED_RETRY_MINUTES=60) and logic.
|
||||
Its behavior is independent of changes to recover_stale_documents().
|
||||
|
||||
**Validates: Requirements 3.7**
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@given(
|
||||
num_failed_docs=st.integers(min_value=1, max_value=100),
|
||||
)
|
||||
@settings(max_examples=100)
|
||||
async def test_retry_failed_extractions_processes_failed_docs(
|
||||
self,
|
||||
num_failed_docs: int,
|
||||
):
|
||||
"""Property: for all documents in extraction_failed status, retry logic unchanged.
|
||||
|
||||
Verify that retry_failed_extractions:
|
||||
1. Queries documents with status='extraction_failed'
|
||||
2. Uses EXTRACTION_FAILED_RETRY_MINUTES (60) as the age threshold
|
||||
3. Enqueues them via _enqueue_if_new
|
||||
4. Resets their status to 'parsed'
|
||||
5. Deletes failed intelligence rows
|
||||
"""
|
||||
from services.scheduler.app import (
|
||||
EXTRACTION_FAILED_RETRY_MINUTES,
|
||||
retry_failed_extractions,
|
||||
)
|
||||
|
||||
# Verify the retry threshold hasn't changed
|
||||
assert EXTRACTION_FAILED_RETRY_MINUTES == 60, (
|
||||
f"EXTRACTION_FAILED_RETRY_MINUTES changed from 60 to "
|
||||
f"{EXTRACTION_FAILED_RETRY_MINUTES} — this should be preserved"
|
||||
)
|
||||
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
old_time = now - timedelta(minutes=EXTRACTION_FAILED_RETRY_MINUTES + 10)
|
||||
|
||||
# Generate fake failed document rows
|
||||
fake_rows = []
|
||||
for i in range(num_failed_docs):
|
||||
fake_rows.append({
|
||||
"id": uuid.uuid4(),
|
||||
"document_type": "news" if i % 3 != 0 else "macro_event",
|
||||
"ticker": f"TICK{i}",
|
||||
"updated_at": old_time,
|
||||
})
|
||||
|
||||
pool = AsyncMock()
|
||||
pool.fetch = AsyncMock(return_value=fake_rows)
|
||||
pool.execute = AsyncMock()
|
||||
|
||||
rds = AsyncMock()
|
||||
rds.set = AsyncMock(return_value=True) # All enqueues succeed
|
||||
rds.rpush = AsyncMock()
|
||||
|
||||
result = await retry_failed_extractions(pool, rds)
|
||||
|
||||
# All docs should be enqueued
|
||||
assert result == num_failed_docs, (
|
||||
f"Expected {num_failed_docs} docs retried, got {result}"
|
||||
)
|
||||
|
||||
# Verify the SQL query used the correct threshold
|
||||
fetch_call = pool.fetch.call_args
|
||||
sql_query = fetch_call[0][0]
|
||||
threshold_param = fetch_call[0][1]
|
||||
|
||||
assert "extraction_failed" in sql_query, (
|
||||
"retry_failed_extractions should query for 'extraction_failed' status"
|
||||
)
|
||||
assert threshold_param == EXTRACTION_FAILED_RETRY_MINUTES, (
|
||||
f"Expected threshold param {EXTRACTION_FAILED_RETRY_MINUTES}, "
|
||||
f"got {threshold_param}"
|
||||
)
|
||||
|
||||
# Verify pool.execute was called for both DELETE and UPDATE
|
||||
assert pool.execute.call_count == 2, (
|
||||
f"Expected 2 pool.execute calls (DELETE + UPDATE), "
|
||||
f"got {pool.execute.call_count}"
|
||||
)
|
||||
|
||||
# Verify the DELETE query targets document_intelligence with failed status
|
||||
delete_call = pool.execute.call_args_list[0]
|
||||
assert "DELETE" in delete_call[0][0] and "document_intelligence" in delete_call[0][0], (
|
||||
"First execute should DELETE from document_intelligence"
|
||||
)
|
||||
|
||||
# Verify the UPDATE resets status to 'parsed'
|
||||
update_call = pool.execute.call_args_list[1]
|
||||
assert "parsed" in update_call[0][0] and "UPDATE" in update_call[0][0], (
|
||||
"Second execute should UPDATE status to 'parsed'"
|
||||
)
|
||||
Reference in New Issue
Block a user