feat: trading feedback engine — periodic performance reports with AI summarization
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled

- Migration 038: trading_reports table + report-summarizer agent seed
- 6 reporting modules: models, collector, sections, validator, summarizer, generator
- API endpoints: GET /api/reports (paginated, filterable), GET /api/reports/{id}
- Frontend hooks: useReports, useReport with TanStack Query
- Scheduler: daily (after 16:30 ET) and weekly (Saturday) report triggers
- Redis queue consumer for async report generation with retry/dedup
- 5 property-based tests (chunking, serialization, validation, accuracy, deltas)
- 109 unit/integration tests across all modules
- 6 frontend hook tests with MSW mocks
This commit is contained in:
Celes Renata
2026-05-01 22:13:09 +00:00
parent 376fcb4bb4
commit bc077bfcc8
28 changed files with 6771 additions and 1 deletions
+175
View File
@@ -0,0 +1,175 @@
"""Report validator — cross-checks computed metrics against live data.
Compares report section values against prediction_outcomes and
model_metric_snapshots, flagging discrepancies that exceed the
configured threshold.
"""
from __future__ import annotations
import logging
import math
from services.reporting.models import (
ModelQualitySection,
RecommendationAccuracySection,
ReportData,
ValidationStatus,
ValidationWarning,
)
logger = logging.getLogger(__name__)
DISCREPANCY_THRESHOLD_PCT = 5.0
def _sanitize(value: float | None) -> float:
"""Replace None, NaN, and infinity with 0.0."""
if value is None:
return 0.0
if math.isnan(value) or math.isinf(value):
return 0.0
return value
def _check_discrepancy(
field_name: str,
computed: float,
snapshot: float,
) -> ValidationWarning | None:
"""Compare computed vs snapshot and return a warning if >5% discrepancy.
Edge cases:
- snapshot=0 and computed≠0 → 100% difference → warning
- both=0 → 0% difference → no warning
- snapshot is handled upstream (NULL → skip before calling this)
"""
computed = _sanitize(computed)
snapshot = _sanitize(snapshot)
if snapshot == 0.0 and computed == 0.0:
return None
if snapshot == 0.0:
# Non-zero computed with zero snapshot → 100% discrepancy
pct_diff = 100.0
else:
pct_diff = abs(computed - snapshot) / abs(snapshot) * 100.0
if pct_diff > DISCREPANCY_THRESHOLD_PCT:
return ValidationWarning(
field_name=field_name,
computed_value=computed,
snapshot_value=snapshot,
pct_difference=round(pct_diff, 4),
)
return None
def validate_recommendation_accuracy(
section: RecommendationAccuracySection,
prediction_outcomes: list[dict],
) -> list[ValidationWarning]:
"""Cross-reference reported win rates with prediction_outcomes.
Computes win_rate from prediction_outcomes (count profitable / total)
and compares against section.acted_win_rate. Returns warnings for
discrepancies > 5%.
"""
warnings: list[ValidationWarning] = []
if not prediction_outcomes:
return warnings
total = len(prediction_outcomes)
profitable_count = sum(
1 for po in prediction_outcomes if po.get("profitable")
)
computed_win_rate = profitable_count / total if total > 0 else 0.0
w = _check_discrepancy(
"acted_win_rate",
section.acted_win_rate,
computed_win_rate,
)
if w is not None:
warnings.append(w)
return warnings
def validate_model_quality(
section: ModelQualitySection,
metric_snapshots: list[dict],
) -> list[ValidationWarning]:
"""Compare reported model quality metrics against model_metric_snapshots.
For each window in the section, finds the matching snapshot by
lookback_window and compares win_rate, directional_accuracy,
information_coefficient, calibration_error, and brier_score.
Flags discrepancies > 5%.
"""
warnings: list[ValidationWarning] = []
if not metric_snapshots:
return warnings
# Build lookup: lookback_window → latest snapshot (first match since
# collector orders by generated_at DESC)
snap_by_window: dict[str, dict] = {}
for snap in metric_snapshots:
window = snap.get("lookback_window", "")
if window and window not in snap_by_window:
snap_by_window[window] = snap
metric_fields = [
("win_rate", "win_rate"),
("directional_accuracy", "directional_accuracy"),
("information_coefficient", "information_coefficient"),
("calibration_error", "calibration_error"),
("brier_score", "brier_score"),
]
for mq_window in section.windows:
snap = snap_by_window.get(mq_window.lookback)
if snap is None:
continue
for section_attr, snap_key in metric_fields:
section_value = getattr(mq_window, section_attr, None)
snapshot_value = snap.get(snap_key)
# NULL snapshot → skip
if snapshot_value is None:
continue
# NULL section value → skip
if section_value is None:
continue
snapshot_float = _sanitize(float(snapshot_value))
section_float = _sanitize(section_value)
w = _check_discrepancy(
f"{mq_window.lookback}_{section_attr}",
section_float,
snapshot_float,
)
if w is not None:
warnings.append(w)
return warnings
def compute_validation_status(report: ReportData) -> ValidationStatus:
"""Determine overall validation status.
Returns 'passed' if no warnings across all sections,
'warnings' if any section has validation warnings.
"""
if report.pnl.validation_warnings:
return ValidationStatus.WARNINGS
if report.recommendation_accuracy.validation_warnings:
return ValidationStatus.WARNINGS
if report.model_quality.validation_warnings:
return ValidationStatus.WARNINGS
return ValidationStatus.PASSED