stonks-oracle/docs/notes/session-context-2026-07-11.md

Session Context — July 11, 2026

Current State

Active Namespace: stonks-beta

• This is the ONLY namespace that should be running
• stonks-oracle namespace has been scaled to 0 replicas (all deployments)
• Dashboard: https://stonks-beta.celestium.life
• API: https://stonks-api-beta.celestium.life

What Was Done This Session

1. Pipeline Health Fixes (spec: .kiro/specs/pipeline-health-fixes/)

All implemented and deployed:

• Stuck Parsed Docs: STALE_PARSED_THRESHOLD_MINUTES 240→30, LIMIT 100→500, _ENQUEUED_TTL 14400→3600 (services/scheduler/app.py)
• Price Fallback: Added 24h market_snapshots time-window fallback in create_prediction_snapshot() (services/validation/prediction_snapshot.py)
• Sentiment Normalization: Added normalize_impact_scores() z-score function (services/aggregation/scoring.py) + integrated into aggregate_company_window() (services/aggregation/worker.py)
• Signal Engine: Replicas set to 0 in all Helm values files
• Quality Gate: max_snapshot_age_hours 24→48 (services/trading/model_quality_gate.py)
• Backfill script: scripts/backfill_snapshot_prices.py (one-time, not yet run on beta)

2. Extractor Null-Field Fix

• services/extractor/schemas.py: _normalize_extraction_data() now handles None values (not just missing keys) and filters out company entries with empty ticker
• Test updated: tests/test_extractor_schemas.py::test_validate_semantic_missing_ticker_is_error

3. Macro Doc Status Fix

• services/extractor/main.py: _process_macro_classification() now updates document status to 'extracted' on success, 'extraction_failed' on error
• Beta DB: manually fixed 1849 stuck macro docs (UPDATE status='extracted' WHERE id IN global_events)

4. Dashboard Fix

• frontend/src/pages/OpsPipeline.tsx: Document Stages now uses time-filtered /health data (consistent with other sections), all-time from SSE stream shown as subtitle, time range labels added to all sections

5. CI/CD DNS Fix

• .woodpecker/*.yml: All 5 pipeline files now use clone.git.settings.remote: http://10.43.73.77:3000/admin/stonks-oracle.git (Gitea ClusterIP directly, bypasses DNS)
• CoreDNS: scaled to 4 replicas, forward . 192.168.42.1, dnsPolicy: None with nameservers: [192.168.42.1]
• Woodpecker: WOODPECKER_BACKEND_K8S_DNS_CONFIG has nameservers:[10.43.0.10] + searches including git-server.svc.cluster.local

Known Issues / TODO

1. stonks-oracle namespace: Scaled to 0 but still exists with stale data (42K extraction queue in Redis DB 0). Could be cleaned up or deleted entirely.

2. Thesis Rewriter agent: Was hammering vLLM from stonks-oracle namespace (5600+ calls/24h). Now stopped since namespace is scaled down. If it was also running in beta, check if recommendation service is calling vLLM for thesis rewrites excessively.

3. AxionML/Qwen3.5-9B-NVFP4 requests: Something external is hitting vLLM with a model that doesn't exist (404s). Not from our pipeline — likely Open WebUI or another tool on the network configured with wrong model name. Source IP: goes through vllm-metrics nginx proxy (10.42.1.155).

4. GitHub mirror: finalize.yml mirror-github step fails (SSH key or DNS). Has failure: ignore so non-blocking. Needs github_ssh_key secret configured in Woodpecker.

5. OpsPipeline dashboard: Numbers now show time-filtered data. The "Document Stages" section shows counts from the selected time window (default 24h), with all-time totals as subtle subtitles. Currently beta shows: extracted=5417, low_quality=1659, parsed=15.

6. Aggregation not generating trends on weekends: Expected — market hours check prevents weekend trend generation. Will resume Monday.

7. 15 docs still in parsed status: These are likely fresh ingests waiting for the next extraction cycle. Not stuck.

Agent Performance (beta, last 24h as of session end)

• Document Intelligence Extractor: 33 calls, 94% success, avg 11.4s, conf 0.794
• Global Event Classifier: 81 calls, 99% success, avg 4.2s, conf 0.745
• Thesis Rewriter: 5603 calls, 100% success, avg 2.5s (from stonks-oracle before shutdown)
• Report Summarizer: 6 calls, 100% success, avg 6.9s

Infrastructure

• k3s cluster: 4 NixOS nodes (gremlin-1 through gremlin-4)
• vLLM: vllm-service namespace, model numind/NuExtract3, 4070 Ti Super 16GB
• CoreDNS: 4 replicas, forward . 192.168.42.1
• Redis: DB 0 = stonks-oracle (stale), DB 1 = stonks-beta (active)
• PostgreSQL: shared instance, both namespaces use same DB server (different databases? or same? — needs verification)
• Gitea: git-server namespace, ClusterIP 10.43.73.77:3000, NodePort 30300
• Woodpecker: woodpecker namespace, kubernetes backend, 2 agents

Key Files Modified

services/scheduler/app.py          — recovery thresholds + batch limit
services/validation/prediction_snapshot.py — 24h price fallback
services/aggregation/scoring.py    — normalize_impact_scores()
services/aggregation/worker.py     — normalization integration
services/trading/model_quality_gate.py — 48h threshold
services/extractor/schemas.py      — null field handling
services/extractor/main.py         — macro doc status update
frontend/src/pages/OpsPipeline.tsx  — dashboard fix
scripts/backfill_snapshot_prices.py — new script
tests/test_pbt_pipeline_health_*.py — PBT tests
tests/test_extractor_schemas.py    — updated test
infra/helm/stonks-oracle/values*.yaml — signal-engine replicas
.woodpecker/*.yml                  — ClusterIP clone fix