Celes Renata
a72f336ad1
Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.
85 lines
5.5 KiB
Markdown
85 lines
5.5 KiB
Markdown
# Session Context — July 11, 2026
|
|
|
|
## Current State
|
|
|
|
### Active Namespace: `stonks-beta`
|
|
- This is the ONLY namespace that should be running
|
|
- `stonks-oracle` namespace has been scaled to 0 replicas (all deployments)
|
|
- Dashboard: `https://stonks-beta.celestium.life`
|
|
- API: `https://stonks-api-beta.celestium.life`
|
|
|
|
### What Was Done This Session
|
|
|
|
#### 1. Pipeline Health Fixes (spec: `.kiro/specs/pipeline-health-fixes/`)
|
|
All implemented and deployed:
|
|
- **Stuck Parsed Docs**: `STALE_PARSED_THRESHOLD_MINUTES` 240→30, `LIMIT` 100→500, `_ENQUEUED_TTL` 14400→3600 (`services/scheduler/app.py`)
|
|
- **Price Fallback**: Added 24h market_snapshots time-window fallback in `create_prediction_snapshot()` (`services/validation/prediction_snapshot.py`)
|
|
- **Sentiment Normalization**: Added `normalize_impact_scores()` z-score function (`services/aggregation/scoring.py`) + integrated into `aggregate_company_window()` (`services/aggregation/worker.py`)
|
|
- **Signal Engine**: Replicas set to 0 in all Helm values files
|
|
- **Quality Gate**: `max_snapshot_age_hours` 24→48 (`services/trading/model_quality_gate.py`)
|
|
- **Backfill script**: `scripts/backfill_snapshot_prices.py` (one-time, not yet run on beta)
|
|
|
|
#### 2. Extractor Null-Field Fix
|
|
- `services/extractor/schemas.py`: `_normalize_extraction_data()` now handles `None` values (not just missing keys) and filters out company entries with empty ticker
|
|
- Test updated: `tests/test_extractor_schemas.py::test_validate_semantic_missing_ticker_is_error`
|
|
|
|
#### 3. Macro Doc Status Fix
|
|
- `services/extractor/main.py`: `_process_macro_classification()` now updates document status to 'extracted' on success, 'extraction_failed' on error
|
|
- Beta DB: manually fixed 1849 stuck macro docs (UPDATE status='extracted' WHERE id IN global_events)
|
|
|
|
#### 4. Dashboard Fix
|
|
- `frontend/src/pages/OpsPipeline.tsx`: Document Stages now uses time-filtered `/health` data (consistent with other sections), all-time from SSE stream shown as subtitle, time range labels added to all sections
|
|
|
|
#### 5. CI/CD DNS Fix
|
|
- `.woodpecker/*.yml`: All 5 pipeline files now use `clone.git.settings.remote: http://10.43.73.77:3000/admin/stonks-oracle.git` (Gitea ClusterIP directly, bypasses DNS)
|
|
- CoreDNS: scaled to 4 replicas, `forward . 192.168.42.1`, `dnsPolicy: None` with `nameservers: [192.168.42.1]`
|
|
- Woodpecker: `WOODPECKER_BACKEND_K8S_DNS_CONFIG` has `nameservers:[10.43.0.10]` + searches including `git-server.svc.cluster.local`
|
|
|
|
### Known Issues / TODO
|
|
|
|
1. **`stonks-oracle` namespace**: Scaled to 0 but still exists with stale data (42K extraction queue in Redis DB 0). Could be cleaned up or deleted entirely.
|
|
|
|
2. **Thesis Rewriter agent**: Was hammering vLLM from stonks-oracle namespace (5600+ calls/24h). Now stopped since namespace is scaled down. If it was also running in beta, check if recommendation service is calling vLLM for thesis rewrites excessively.
|
|
|
|
3. **`AxionML/Qwen3.5-9B-NVFP4` requests**: Something external is hitting vLLM with a model that doesn't exist (404s). Not from our pipeline — likely Open WebUI or another tool on the network configured with wrong model name. Source IP: goes through `vllm-metrics` nginx proxy (`10.42.1.155`).
|
|
|
|
4. **GitHub mirror**: `finalize.yml` mirror-github step fails (SSH key or DNS). Has `failure: ignore` so non-blocking. Needs `github_ssh_key` secret configured in Woodpecker.
|
|
|
|
5. **OpsPipeline dashboard**: Numbers now show time-filtered data. The "Document Stages" section shows counts from the selected time window (default 24h), with all-time totals as subtle subtitles. Currently beta shows: extracted=5417, low_quality=1659, parsed=15.
|
|
|
|
6. **Aggregation not generating trends on weekends**: Expected — market hours check prevents weekend trend generation. Will resume Monday.
|
|
|
|
7. **15 docs still in `parsed` status**: These are likely fresh ingests waiting for the next extraction cycle. Not stuck.
|
|
|
|
### Agent Performance (beta, last 24h as of session end)
|
|
- Document Intelligence Extractor: 33 calls, 94% success, avg 11.4s, conf 0.794
|
|
- Global Event Classifier: 81 calls, 99% success, avg 4.2s, conf 0.745
|
|
- Thesis Rewriter: 5603 calls, 100% success, avg 2.5s (from stonks-oracle before shutdown)
|
|
- Report Summarizer: 6 calls, 100% success, avg 6.9s
|
|
|
|
### Infrastructure
|
|
- k3s cluster: 4 NixOS nodes (gremlin-1 through gremlin-4)
|
|
- vLLM: `vllm-service` namespace, model `numind/NuExtract3`, 4070 Ti Super 16GB
|
|
- CoreDNS: 4 replicas, `forward . 192.168.42.1`
|
|
- Redis: DB 0 = stonks-oracle (stale), DB 1 = stonks-beta (active)
|
|
- PostgreSQL: shared instance, both namespaces use same DB server (different databases? or same? — needs verification)
|
|
- Gitea: `git-server` namespace, ClusterIP 10.43.73.77:3000, NodePort 30300
|
|
- Woodpecker: `woodpecker` namespace, kubernetes backend, 2 agents
|
|
|
|
### Key Files Modified
|
|
```
|
|
services/scheduler/app.py — recovery thresholds + batch limit
|
|
services/validation/prediction_snapshot.py — 24h price fallback
|
|
services/aggregation/scoring.py — normalize_impact_scores()
|
|
services/aggregation/worker.py — normalization integration
|
|
services/trading/model_quality_gate.py — 48h threshold
|
|
services/extractor/schemas.py — null field handling
|
|
services/extractor/main.py — macro doc status update
|
|
frontend/src/pages/OpsPipeline.tsx — dashboard fix
|
|
scripts/backfill_snapshot_prices.py — new script
|
|
tests/test_pbt_pipeline_health_*.py — PBT tests
|
|
tests/test_extractor_schemas.py — updated test
|
|
infra/helm/stonks-oracle/values*.yaml — signal-engine replicas
|
|
.woodpecker/*.yml — ClusterIP clone fix
|
|
```
|