admin/stonks-oracle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implement dual-pipeline signal engine service
ci/woodpecker/push/test Pipeline was successful
Details
ci/woodpecker/push/build-2 Pipeline was successful
Details
ci/woodpecker/push/build-1 Pipeline was successful
Details
ci/woodpecker/push/build-3 Pipeline was successful
Details
ci/woodpecker/push/finalize Pipeline was successful
Details
Build and Push / lint-and-test (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Details
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Details
Build and Push / build-dashboard (push) Has been cancelled
Details
Build and Push / build-superset (push) Has been cancelled
Details
Build and Push / integration-test (push) Has been cancelled
Details
Build and Push / beta-gate (push) Has been cancelled
Details

Browse Source 
New service at services/signal_engine/ implementing concurrent heuristic
(deterministic scoring) and probabilistic (Bayesian inference) pipelines
that evaluate technical signals across 6 timeframes (M30-M) and produce
independent BUY/WATCH/SKIP verdicts per ticker per evaluation tick.

Components:
- Input Normalizer: multi-source data assembly with sentinel fallbacks
- Signal Library: Fibonacci, MA Stack, RSI, Cup & Handle, Elliott Wave
- Multi-Timeframe Confluence Engine: weighted scoring with D/W/M anchors
- Hard Filter Engine: macro_bias, valuation, earnings proximity gating
- Heuristic Pipeline: S_total scoring with confidence-gated verdicts
- Probabilistic Pipeline: Bayesian log-odds with regime priors, entropy
  gating, EV_R calculation, and signal correlation penalty
- Exit Engine: stop-loss, targets, trailing ATR-based stops
- Delta Analyzer: pipeline agreement tracking with rolling Redis metrics
- Output Formatter: SignalOutput contract + Recommendation schema mapping
- Worker orchestrator: concurrent pipelines with failure isolation
- Main entry point: queue polling with fail-safe config loading

Infrastructure:
- Migration 039: signal_engine_outputs table with 3 indexes
- Helm chart: signalEngine service entry (processing tier)
- Redis key: QUEUE_SIGNAL_ENGINE constant

Tests: 390 tests (unit + property-based) covering all components
Config: dual_pipeline_enabled=false by default (safe rollout)
This commit is contained in:
Celes Renata
2026-05-02 07:32:26 +00:00
parent 7e2343ec2c
commit f468e30af0
61 changed files with 14107 additions and 184 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/helm-reference.md
+48 -31
View File
@@ -94,7 +94,7 @@ Each key under `services` defines a Kubernetes Deployment. The deployments templ
| `image` | string | yes | Image name appended to `image.registry`. Also used as the Deployment name and pod label (`app: <image>`). |
| `command` | string | no | Shell command passed as `["sh", "-c", "<command>"]`. Omit for images with a built-in entrypoint (e.g., dashboard/nginx). |
| `tier` | string | yes | Service tier label (`stonks-oracle/tier`). One of: `api`, `frontend`, `processing`, `trading`, `orchestration`, `analytics`, `ingestion`. |
| `port` | int | no | Container port. When set, a Kubernetes Service is created mapping `port → port`. |
| `port` | int | no | Container port. When set, a Kubernetes Service is created mapping `port -> port`. |
| `pipeline` | bool | no | If `true`, replicas are set to 0 when `pipelineEnabled` is `false`. |
| `secrets` | list(string) | no | List of Secret names to mount via `envFrom.secretRef`. |
| `resources` | object | yes | Kubernetes resource requests and limits (`cpu`, `memory`). |
@@ -118,9 +118,10 @@ Each key under `services` defines a Kubernetes Deployment. The deployments templ
| `resources.limits` | cpu: 200m, memory: 128Mi |
| `probes` | — |
The scheduler deployment has two init containers (not configurable via values):
The scheduler deployment has three init containers (not configurable via values):
1. **run-migrations** — applies all SQL files from `infra/migrations/*.sql` in sorted order.
2. **seed-if-empty** — runs `python -m services.symbol_registry.seed` if the `companies` table is empty.
3. **backfill-market-data** — runs `scripts/backfill_market_data.py` if available (skips gracefully if not).
#### symbolRegistry
@@ -141,7 +142,7 @@ The scheduler deployment has two init containers (not configurable via values):
| Field | Value |
|-------|-------|
| `replicas` | `2` |
| `replicas` | `1` |
| `pipeline` | `true` |
| `image` | `ingestion` |
| `command` | `python -m services.ingestion.worker` |
@@ -274,7 +275,7 @@ Single replica is recommended — the extractor is bottlenecked by the shared Ol
| `command` | `uvicorn services.api.app:app --host 0.0.0.0 --port 8000` |
| `tier` | `api` |
| `port` | `8000` |
| `secrets` | `stonks-core-secrets` |
| `secrets` | `stonks-core-secrets`, `stonks-market-secrets` |
| `resources.requests` | cpu: 100m, memory: 128Mi |
| `resources.limits` | cpu: 500m, memory: 256Mi |
| `probes.readiness` | path: `/docs`, port: 8000, initialDelay: 5s, period: 10s |
@@ -323,7 +324,7 @@ All keys under `config` are rendered into a Kubernetes ConfigMap named `stonks-c
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `config.OLLAMA_BASE_URL` | string | `""` (empty) | Ollama API base URL. Set to the cluster-internal or external Ollama endpoint. |
| `config.OLLAMA_BASE_URL` | string | `http://10.1.1.12:2701` | Ollama API base URL. Points to the external Ollama endpoint by default. |
| `config.OLLAMA_MODEL` | string | `qwen3.5:9b-fast` | Default LLM model for extraction and classification agents. |
| `config.OLLAMA_TIMEOUT` | string | `240` | Request timeout in seconds for Ollama API calls. |
| `config.OLLAMA_MAX_RETRIES` | string | `2` | Maximum retry attempts for failed Ollama requests. |
@@ -331,6 +332,17 @@ All keys under `config` are rendered into a Kubernetes ConfigMap named `stonks-c
| `config.OLLAMA_RETRY_MAX_DELAY` | string | `10.0` | Maximum delay cap in seconds for Ollama retry backoff. |
| `config.OLLAMA_RETRY_BACKOFF_MULTIPLIER` | string | `2.0` | Multiplier for exponential backoff between Ollama retries. |
### vLLM
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `config.VLLM_BASE_URL` | string | `http://10.1.1.12:2701` | vLLM API base URL. Alternative LLM backend using OpenAI-compatible API. |
| `config.VLLM_MODEL` | string | `qwen3.5:9b-fast` | vLLM model identifier. |
| `config.VLLM_TIMEOUT` | string | `120` | Request timeout in seconds for vLLM API calls. |
| `config.VLLM_MAX_RETRIES` | string | `2` | Maximum retry attempts for failed vLLM requests. |
| `config.VLLM_TEMPERATURE` | string | `0.7` | Sampling temperature for vLLM generation (0.0-1.0). |
| `config.VLLM_API_KEY` | string | `""` (empty) | API key for vLLM authentication. Leave empty if not required. |
### Analytics / Trino
| Key | Type | Default | Description |
@@ -347,7 +359,7 @@ All keys under `config` are rendered into a Kubernetes ConfigMap named `stonks-c
|-----|------|---------|-------------|
| `config.BROKER_MODE` | string | `paper` | Broker execution mode. `paper` for simulated trading, `live` for real orders. |
| `config.BROKER_PROVIDER` | string | `""` (empty) | Broker provider name (e.g., `alpaca`). |
| `config.MARKET_DATA_BASE_URL` | string | `""` (empty) | Market data API base URL (e.g., `https://api.polygon.io`). |
| `config.MARKET_DATA_BASE_URL` | string | `https://api.polygon.io` | Market data API base URL. |
| `config.MARKET_DATA_PROVIDER` | string | `polygon` | Market data provider identifier. |
| `config.TRADING_ENABLED` | string | `true` | Master toggle for the trading engine. Set to `false` to disable order submission. |
| `config.TRADING_RISK_TIER` | string | `moderate` | Default risk tier for position sizing. Options: `conservative`, `moderate`, `aggressive`. |
@@ -384,7 +396,7 @@ All keys under `config` are rendered into a Kubernetes ConfigMap named `stonks-c
|-----|------|---------|-------------|
| `config.ALERT_SOURCE_FAILURE_THRESHOLD` | string | `3` | Number of consecutive source failures before firing an alert. |
| `config.ALERT_SOURCE_FAILURE_WINDOW_HOURS` | string | `6` | Time window (hours) for evaluating source failure count. |
| `config.ALERT_SCHEMA_FAILURE_RATE_THRESHOLD` | string | `0.3` | Schema validation failure rate (0.0–1.0) that triggers an alert. |
| `config.ALERT_SCHEMA_FAILURE_RATE_THRESHOLD` | string | `0.3` | Schema validation failure rate (0.0-1.0) that triggers an alert. |
| `config.ALERT_SCHEMA_FAILURE_WINDOW_HOURS` | string | `1` | Time window (hours) for evaluating schema failure rate. |
| `config.ALERT_LAKE_LAG_THRESHOLD_MINUTES` | string | `60` | Minutes of lakehouse publish lag before alerting. |
| `config.ALERT_BROKER_ERROR_THRESHOLD` | string | `3` | Number of broker errors before firing an alert. |
@@ -395,7 +407,7 @@ All keys under `config` are rendered into a Kubernetes ConfigMap named `stonks-c
## `secrets` — Kubernetes Secrets
Secrets are rendered into five Kubernetes Secret objects. In the base `values.yaml`, all secret values default to empty strings. Inject real values at deploy time using `--set` flags or a values override file.
Secrets are rendered into five Kubernetes Secret objects. Inject real values at deploy time using `--set` flags or a values override file. The base `values.yaml` contains placeholder values — override them for each environment.
### Secret Objects
@@ -403,32 +415,32 @@ Secrets are rendered into five Kubernetes Secret objects. In the base `values.ya
|-------------|-----------|-------------|
| `stonks-core-secrets` | `secrets.core` | All services |
| `stonks-broker-secrets` | `secrets.broker` | ingestion, trading-engine, risk-engine, broker-adapter |
| `stonks-market-secrets` | `secrets.market` | ingestion |
| `stonks-market-secrets` | `secrets.market` | ingestion, query-api |
| `stonks-gmail-secrets` | `secrets.gmail` | trading-engine |
| `stonks-dashboard-secrets` | `secrets.dashboard` | superset |
### `secrets.core`
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `POSTGRES_PASSWORD` | string | `""` | PostgreSQL password. |
| `MINIO_ACCESS_KEY` | string | `""` | MinIO access key (AWS-style). |
| `MINIO_SECRET_KEY` | string | `""` | MinIO secret key. |
| `REDIS_PASSWORD` | string | `""` | Redis authentication password. |
| Key | Type | Description |
|-----|------|-------------|
| `POSTGRES_PASSWORD` | string | PostgreSQL password. |
| `MINIO_ACCESS_KEY` | string | MinIO access key (AWS-style). |
| `MINIO_SECRET_KEY` | string | MinIO secret key. |
| `REDIS_PASSWORD` | string | Redis authentication password. |
### `secrets.broker`
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `BROKER_API_KEY` | string | `""` | Broker API key (e.g., Alpaca paper trading key). |
| `BROKER_API_SECRET` | string | `""` | Broker API secret. |
| `BROKER_BASE_URL` | string | `""` | Broker API base URL (e.g., `https://paper-api.alpaca.markets`). |
| Key | Type | Description |
|-----|------|-------------|
| `BROKER_API_KEY` | string | Broker API key (e.g., Alpaca paper trading key). |
| `BROKER_API_SECRET` | string | Broker API secret. |
| `BROKER_BASE_URL` | string | Broker API base URL (e.g., `https://paper-api.alpaca.markets`). |
### `secrets.market`
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `MARKET_DATA_API_KEY` | string | `""` | Market data provider API key (e.g., Polygon.io). |
| Key | Type | Description |
|-----|------|-------------|
| `MARKET_DATA_API_KEY` | string | Market data provider API key (e.g., Polygon.io). |
### `secrets.gmail`
@@ -440,10 +452,10 @@ Secrets are rendered into five Kubernetes Secret objects. In the base `values.ya
### `secrets.dashboard`
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `SUPERSET_SECRET_KEY` | string | `""` | Flask secret key for Superset session encryption. |
| `SUPERSET_ADMIN_PASSWORD` | string | `""` | Superset admin user password. |
| Key | Type | Description |
|-----|------|-------------|
| `SUPERSET_SECRET_KEY` | string | Flask secret key for Superset session encryption. |
| `SUPERSET_ADMIN_PASSWORD` | string | Superset admin user password. |
### Injecting Secrets at Deploy Time
@@ -596,15 +608,20 @@ Key overrides:
| `pipelineEnabled` | `true` | Services deployed (ArgoCD health checks), but pipeline defaults to OFF via `PIPELINE_DEFAULT_OFF`. |
| `config.DEPLOY_STAGE` | `beta` | Isolates Redis keys (`stonks:beta:*`) and MinIO buckets (`beta-stonks-*`). |
| `config.POSTGRES_DB` | `stonks_beta` | Separate database for beta data. |
| `config.POSTGRES_USER` | `stonks_beta` | Separate database user for beta. |
| `config.REDIS_DB` | `1` | Separate Redis DB index. |
| `config.LOG_LEVEL` | `DEBUG` | Verbose logging for debugging. |
| `config.TRADING_ENABLED` | `false` | Safety net — no order submission in beta. |
| `config.PIPELINE_DEFAULT_OFF` | `true` | Scheduler won't enqueue jobs unless explicitly enabled. |
| `config.TRADING_ENABLED` | `true` | Trading engine active but constrained by paper broker mode. |
| `config.PIPELINE_DEFAULT_OFF` | `true` | Scheduler won't enqueue jobs unless explicitly enabled via the UI. |
| `config.BROKER_MODE` | `paper` | Simulated order execution. |
| `config.BROKER_PROVIDER` | `alpaca` | Alpaca paper trading API. |
| `config.OLLAMA_MODEL` | `qwen3.6` | May use a different model version for testing. |
| `trino.enabled` | `false` | Analytics stack disabled in beta. |
| `hiveMetastore.enabled` | `false` | Analytics stack disabled in beta. |
| `superset.enabled` | `false` | Analytics stack disabled in beta. |
Beta also configures vLLM settings (`VLLM_BASE_URL`, `VLLM_MODEL`, etc.) for testing alternative LLM backends.
Beta ingress hostnames:
| Service | Hostname |
@@ -649,11 +666,11 @@ Paper ingress hostnames:
```
values-beta.yaml values-paper.yaml values.yaml (base)
Beta → Paper Trading → Production
Beta -> Paper Trading -> Production
Integration Simulated orders Live trading
testing Real market data Real orders
Pipeline OFF Pipeline ON Pipeline ON
Trading OFF Trading ON Trading ON
Trading ON Trading ON Trading ON
Analytics OFF Analytics ON Analytics ON
```