stonks-oracle/.kiro/specs/pipeline-health-fixes/tasks.md

Implementation Plan

Overview

Bugfix implementation for five pipeline health issues: stuck parsed docs, missing price fallback, uncalibrated sentiment scores, idle signal-engine pods, and overly strict quality gate threshold. Tasks follow the exploratory bugfix workflow: explore bugs via tests, preserve existing behavior, implement fixes, validate.

Tasks

• 1. Write bug condition exploration test

  • Property 1: Bug Condition - Pipeline Health Degradation
  • CRITICAL: This test MUST FAIL on unfixed code - failure confirms the bugs exist
  • DO NOT attempt to fix the test or the code when it fails
  • NOTE: This test encodes the expected behavior - it will validate the fix when it passes after implementation
  • GOAL: Surface counterexamples that demonstrate all five bugs exist
  • Scoped PBT Approach: Scope properties to the concrete failing cases for each bug condition
  • Test file: tests/test_pbt_pipeline_health_bug_condition.py
  • Bug 1 - Batch Overflow: Create 600 documents in parsed status older than 30 min, run recover_stale_documents(), assert up to 500 are recovered per cycle (will FAIL on unfixed code which caps at 100)
  • Bug 2 - Price Fallback Gap: Call create_prediction_snapshot() for ticker with no position and no exact market_snapshots match but data within 24h exists, assert price_at_prediction is NOT NULL (will FAIL on unfixed code which returns NULL)
  • Bug 3 - Sentiment Bias: Generate 50 impact records with systematic negative bias (mean=-0.3, stddev=0.1), compute weighted signals via aggregation, assert normalized output is zero-centered (will FAIL on unfixed code which passes raw scores)
  • Bug 5 - Quality Gate Staleness: Set most recent metric snapshot to 26h ago, evaluate quality gate, assert it passes (will FAIL on unfixed code which rejects at 24h)
  • Run tests on UNFIXED code
  • EXPECTED OUTCOME: Tests FAIL (this is correct - it proves the bugs exist)
  • Document counterexamples: batch capped at 100, price stored as NULL, sentiment heavily negative, quality gate returns passed=False
  • Mark task complete when tests are written, run, and failures are documented
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.6

• 2. Write preservation property tests (BEFORE implementing fix)

  • Property 2: Preservation - Pipeline Behavior Unchanged for Non-Bug Inputs
  • IMPORTANT: Follow observation-first methodology
  • Test file: tests/test_pbt_pipeline_health_preservation.py
  • Normal Doc Processing: Observe that documents < 30 min old are never touched by recover_stale_documents() on unfixed code. Write property: for all documents with age < 30 min, recovery task does NOT enqueue them.
  • Primary Price Path: Observe that when fetch_latest_close_price() returns a valid price, no fallback is invoked. Write property: for all tickers where primary price exists, result equals primary price.
  • Balanced Sentiment: Observe that tickers with symmetric impact_score distributions (mean ≈ 0) produce neutral signals. Write property: for all impact_score sets with mean ≈ 0, normalized output remains centered around 0.
  • Quality Gate Normal: Observe that snapshots < 48h old meeting thresholds pass the quality gate. Write property: for all snapshot ages in [0, 48h) meeting metric criteria, quality gate returns passed=True.
  • Failed Extraction Independence: Observe retry_failed_extractions() behavior is unaffected. Write property: for all documents in extraction_failed status, retry logic unchanged.
  • Run tests on UNFIXED code
  • EXPECTED OUTCOME: Tests PASS (this confirms baseline behavior to preserve)
  • Mark task complete when tests are written, run, and passing on unfixed code
  • Requirements: 3.1, 3.2, 3.3, 3.5, 3.6, 3.7

• 3. Fix: Signal Engine Scale Down (Helm values)

  • 3.1 Set signal-engine replicas to 0 in all Helm values files
    • In infra/helm/stonks-oracle/values.yaml: change signalEngine.replicas from 1 to 0
    • In infra/helm/stonks-oracle/values-beta.yaml: add/set signalEngine.replicas: 0 under services:
    • In infra/helm/stonks-oracle/values-paper.yaml: add/set signalEngine.replicas: 0 under services:
    • Bug_Condition: input.signalEngineReplicas > 0 AND input.dualPipelineEnabled == FALSE
    • Expected_Behavior: signalEngine.replicas == 0 when dual pipeline disabled
    • Preservation: Signal-engine architecture remains functional when re-enabled with replicas > 0
    • Requirements: 2.5, 3.4

• 4. Fix: Quality Gate Threshold Relaxation

  • 4.1 Change max_snapshot_age_hours default from 24 to 48
    • File: services/trading/model_quality_gate.py
    • In QualityGateConfig class, change max_snapshot_age_hours: int = 24 to max_snapshot_age_hours: int = 48
    • Bug_Condition: input.snapshotAgeHours > 24 AND input.snapshotAgeHours <= 48 AND input.maxSnapshotAgeConfig == 24
    • Expected_Behavior: Quality gate accepts snapshots up to 48h old
    • Preservation: Snapshots < 48h meeting criteria continue to promote to live_eligible
    • Requirements: 2.6, 3.5

• 5. Fix: Stuck Parsed Docs Recovery

  • 5.1 Lower STALE_PARSED_THRESHOLD_MINUTES from 240 to 30

    • File: services/scheduler/app.py
    • Change constant: STALE_PARSED_THRESHOLD_MINUTES = 30
    • Documents stuck longer than 30 minutes are likely orphaned
    • Requirements: 2.1

  • 5.2 Increase recovery batch LIMIT from 100 to 500

    • File: services/scheduler/app.py
    • In recover_stale_documents() SQL query, change LIMIT 100 to LIMIT 500
    • Requirements: 2.1

  • 5.3 Update _ENQUEUED_TTL from 14400 to 3600

    • File: services/scheduler/app.py
    • Change _ENQUEUED_TTL = 3600 (1 hour, matching the new recovery cadence)
    • Bug_Condition: input.stuckParsedDocCount > 100 AND input.recoveryBatchLimit == 100 AND input.docStaleMinutes >= 240
    • Expected_Behavior: Recovery processes up to 500 docs per cycle with 30-min threshold
    • Preservation: Documents < 30 min old left alone for extraction queue consumer
    • Requirements: 2.1, 3.1, 3.7

• 6. Fix: Extended Price Fallback

  • 6.1 Add market_snapshots 24h time-window fallback to create_prediction_snapshot()

    • File: services/validation/prediction_snapshot.py
    • After positions fallback fails, query: SELECT close FROM market_snapshots WHERE ticker = $1 AND timestamp >= NOW() - INTERVAL '24 hours' ORDER BY timestamp DESC LIMIT 1
    • Add info-level logging when extended fallback is used
    • Bug_Condition: input.tickerPrice IS NULL AND input.positionPrice IS NULL AND input.marketSnapshotWithin24h IS NOT NULL
    • Expected_Behavior: Use market_snapshots bar close price as price_at_prediction
    • Preservation: Primary fetch_latest_close_price() path unchanged when it returns a valid price
    • Requirements: 2.2, 3.2

  • 6.2 Create backfill script scripts/backfill_snapshot_prices.py

    • Query all prediction_snapshots where price_at_prediction IS NULL
    • For each, attempt: market_snapshots within 24h of generated_at, then positions table
    • Update row with found price
    • Report statistics: found via market_snapshots, found via positions, still NULL
    • Requirements: 2.3

• 7. Fix: Sentiment Z-Score Normalization

  • 7.1 Add normalize_impact_scores() function

    • File: services/aggregation/scoring.py (new helper function)
    • Query 7-day mean and stddev per ticker: SELECT AVG(impact_score) as mean, STDDEV(impact_score) as stddev FROM document_impact_records WHERE ticker = $1 AND created_at >= NOW() - INTERVAL '7 days'
    • Formula: normalized = (raw - mean_7d) / max(stddev_7d, 0.1)
    • Fallback: if fewer than 5 records in 7-day window, return raw score unchanged
    • The 0.1 floor prevents division by near-zero stddev for low-activity tickers
    • Requirements: 2.4

  • 7.2 Integrate normalization into aggregation loop

    • File: services/aggregation/worker.py
    • Before compute_signal_weight() call (around line 440), normalize imp.impact_score via normalize_impact_scores()
    • Pass normalized value into compute_signal_weight() and WeightedSignal
    • Bug_Condition: input.impactScoreUsedRaw == TRUE AND input.ticker7dStddev > 0
    • Expected_Behavior: Normalized impact scores with mean ≈ 0, stddev ≈ 1 for active tickers
    • Preservation: Tickers with balanced distributions continue to produce neutral signals
    • Requirements: 2.4, 3.3

• 8. Verify fixes pass all tests

  • 8.1 Verify bug condition exploration test now passes

    • Property 1: Expected Behavior - Pipeline Health Bugs Resolved
    • IMPORTANT: Re-run the SAME test from task 1 - do NOT write a new test
    • The test from task 1 encodes the expected behavior for all five bugs
    • Run tests/test_pbt_pipeline_health_bug_condition.py
    • EXPECTED OUTCOME: Test PASSES (confirms bugs are fixed)
    • Requirements: 2.1, 2.2, 2.4, 2.6

  • 8.2 Verify preservation tests still pass

    • Property 2: Preservation - Pipeline Behavior Unchanged for Non-Bug Inputs
    • IMPORTANT: Re-run the SAME tests from task 2 - do NOT write new tests
    • Run tests/test_pbt_pipeline_health_preservation.py
    • EXPECTED OUTCOME: Tests PASS (confirms no regressions)
    • Confirm all preservation properties still hold after fixes
    • Requirements: 3.1, 3.2, 3.3, 3.5, 3.6, 3.7

• 9. Lint and final validation

  • Run .venv/bin/ruff check services/ and fix any lint errors
  • Run .venv/bin/python -m pytest tests/ -x --tb=short -q to confirm full test suite passes
  • Verify Helm template renders correctly with 0 signal-engine replicas
  • Requirements: all

• 10. Checkpoint - Ensure all tests pass

  • Ensure all tests pass, ask the user if questions arise.

Task Dependency Graph

{
  "waves": [
    {"tasks": ["1", "2"]},
    {"tasks": ["3", "4"]},
    {"tasks": ["5"]},
    {"tasks": ["6"]},
    {"tasks": ["7"]},
    {"tasks": ["8"]},
    {"tasks": ["9"]},
    {"tasks": ["10"]}
  ]
}

Tasks 3, 4 are independent config changes (no code deps). Task 5 is independent but task 6 builds on the price fallback concept. Task 7 is the most complex (new function + integration). Tasks 8-10 must run after all fixes are applied.

Notes

• Bug 4 (signal engine) is validated by Helm template rendering, not a unit test
• The backfill script (6.2) is a one-time operation, not covered by recurring tests
• Preservation tests use Hypothesis with @settings(max_examples=100) per project conventions
• Test files follow test_pbt_* naming convention per project standards