fix: auto-backfill market data on startup + fix trading dedup bug
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-3 Pipeline was successful
ci/woodpecker/push/build-2 Pipeline was successful
ci/woodpecker/push/build-1 Pipeline was successful
ci/woodpecker/push/finalize Pipeline was successful
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
Build and Push / beta-gate (push) Has been cancelled
1. Add backfill-market-data init container to scheduler deployment. On vanilla start (< 50 bars in market_snapshots), fetches 90 days of daily OHLCV from Polygon directly via asyncpg + httpx. 2. Fix trading engine dedup key placement. Previously the Redis dedup key was set BEFORE evaluate_recommendation(), so recs skipped as outside_trading_window were permanently deduped and never retried when the market opened. Now the dedup key is only set AFTER the decision for non-retryable outcomes (act, confidence_too_low, etc). outside_trading_window skips are retryable.
This commit is contained in:
@@ -778,10 +778,6 @@ class TradingEngine:
|
||||
continue
|
||||
|
||||
# --- Buy path ---
|
||||
# Set dedup key for buys
|
||||
if self.redis is not None:
|
||||
await self.redis.set(trading_dedupe_key(rec_id), "1", ex=86400)
|
||||
|
||||
# Check if we already hold this ticker — don't double up
|
||||
try:
|
||||
existing_pos = await self.pool.fetchrow(
|
||||
@@ -789,6 +785,9 @@ class TradingEngine:
|
||||
ticker,
|
||||
)
|
||||
if existing_pos:
|
||||
# Permanent skip — safe to dedup
|
||||
if self.redis is not None:
|
||||
await self.redis.set(trading_dedupe_key(rec_id), "1", ex=86400)
|
||||
continue
|
||||
except Exception:
|
||||
pass
|
||||
@@ -864,6 +863,19 @@ class TradingEngine:
|
||||
# Persist decision
|
||||
await self._persist_decision(decision)
|
||||
|
||||
# Set dedup key only for permanent outcomes (act or
|
||||
# non-retryable skips). Do NOT dedup
|
||||
# outside_trading_window — those should be retried
|
||||
# when the market opens.
|
||||
retryable_skips = {"outside_trading_window"}
|
||||
if decision.skip_reason not in retryable_skips:
|
||||
if self.redis is not None:
|
||||
await self.redis.set(
|
||||
trading_dedupe_key(rec_id), "1", ex=86400,
|
||||
)
|
||||
if rec_id:
|
||||
self.processed_recommendation_ids.add(rec_id)
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error evaluating recommendation %s", rec.get("recommendation_id", "?"))
|
||||
|
||||
|
||||
Reference in New Issue
Block a user