fix: auto-backfill market data on startup + fix trading dedup bug
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled

1. Add backfill-market-data init container to scheduler deployment.
   On vanilla start (< 50 bars in market_snapshots), fetches 90 days
   of daily OHLCV from Polygon directly via asyncpg + httpx.

2. Fix trading engine dedup key placement. Previously the Redis dedup
   key was set BEFORE evaluate_recommendation(), so recs skipped as
   outside_trading_window were permanently deduped and never retried
   when the market opened. Now the dedup key is only set AFTER the
   decision for non-retryable outcomes (act, confidence_too_low, etc).
   outside_trading_window skips are retryable.
This commit is contained in:
Celes Renata
2026-04-30 14:08:54 +00:00
parent a5f2bcde55
commit 414f476620
2 changed files with 90 additions and 4 deletions
+16 -4
View File
@@ -778,10 +778,6 @@ class TradingEngine:
continue
# --- Buy path ---
# Set dedup key for buys
if self.redis is not None:
await self.redis.set(trading_dedupe_key(rec_id), "1", ex=86400)
# Check if we already hold this ticker — don't double up
try:
existing_pos = await self.pool.fetchrow(
@@ -789,6 +785,9 @@ class TradingEngine:
ticker,
)
if existing_pos:
# Permanent skip — safe to dedup
if self.redis is not None:
await self.redis.set(trading_dedupe_key(rec_id), "1", ex=86400)
continue
except Exception:
pass
@@ -864,6 +863,19 @@ class TradingEngine:
# Persist decision
await self._persist_decision(decision)
# Set dedup key only for permanent outcomes (act or
# non-retryable skips). Do NOT dedup
# outside_trading_window — those should be retried
# when the market opens.
retryable_skips = {"outside_trading_window"}
if decision.skip_reason not in retryable_skips:
if self.redis is not None:
await self.redis.set(
trading_dedupe_key(rec_id), "1", ex=86400,
)
if rec_id:
self.processed_recommendation_ids.add(rec_id)
except Exception:
logger.exception("Error evaluating recommendation %s", rec.get("recommendation_id", "?"))