fix: pipeline stop now halts all workers and flushes queues

Workers (ingestion, parser, extractor, aggregation, recommendation,
broker, lake-publisher) now check the pipeline:enabled Redis flag on
each loop iteration and sleep when disabled.

The toggle endpoint flushes all pipeline queues on disable so queued
jobs don't resume when workers eventually check. Broker/trading queues
are excluded from flush to avoid dropping in-flight orders.

services/recommendation/main.py

@@ -12,7 +12,7 @@ from services.recommendation.worker import generate_recommendation
 from services.shared.agent_config import AgentConfigResolver
 from services.shared.config import OllamaConfig, load_config
 from services.shared.logging import setup_logging
-from services.shared.redis_keys import QUEUE_RECOMMENDATION, queue_key
+from services.shared.redis_keys import QUEUE_RECOMMENDATION, is_pipeline_enabled, queue_key
 
 logger = logging.getLogger("recommendation_main")
 
@@ -62,6 +62,10 @@ async def main() -> None:
 
     try:
         while True:
+            if not await is_pipeline_enabled(redis_client):
+                await asyncio.sleep(1)
+                continue
+
             raw = await redis_client.lpop(queue)
             if raw is None:
                 await asyncio.sleep(1)