# Session Context — July 11, 2026

## Current State

### Active Namespace: `stonks-beta`
- This is the ONLY namespace that should be running
- `stonks-oracle` namespace has been scaled to 0 replicas (all deployments)
- Dashboard: `https://stonks-beta.celestium.life`
- API: `https://stonks-api-beta.celestium.life`

### What Was Done This Session

#### 1. Pipeline Health Fixes (spec: `.kiro/specs/pipeline-health-fixes/`)
All implemented and deployed:
- **Stuck Parsed Docs**: `STALE_PARSED_THRESHOLD_MINUTES` 240→30, `LIMIT` 100→500, `_ENQUEUED_TTL` 14400→3600 (`services/scheduler/app.py`)
- **Price Fallback**: Added 24h market_snapshots time-window fallback in `create_prediction_snapshot()` (`services/validation/prediction_snapshot.py`)
- **Sentiment Normalization**: Added `normalize_impact_scores()` z-score function (`services/aggregation/scoring.py`) + integrated into `aggregate_company_window()` (`services/aggregation/worker.py`)
- **Signal Engine**: Replicas set to 0 in all Helm values files
- **Quality Gate**: `max_snapshot_age_hours` 24→48 (`services/trading/model_quality_gate.py`)
- **Backfill script**: `scripts/backfill_snapshot_prices.py` (one-time, not yet run on beta)

#### 2. Extractor Null-Field Fix
- `services/extractor/schemas.py`: `_normalize_extraction_data()` now handles `None` values (not just missing keys) and filters out company entries with empty ticker
- Test updated: `tests/test_extractor_schemas.py::test_validate_semantic_missing_ticker_is_error`

#### 3. Macro Doc Status Fix
- `services/extractor/main.py`: `_process_macro_classification()` now updates document status to 'extracted' on success, 'extraction_failed' on error
- Beta DB: manually fixed 1849 stuck macro docs (UPDATE status='extracted' WHERE id IN global_events)

#### 4. Dashboard Fix
- `frontend/src/pages/OpsPipeline.tsx`: Document Stages now uses time-filtered `/health` data (consistent with other sections), all-time from SSE stream shown as subtitle, time range labels added to all sections

#### 5. CI/CD DNS Fix
- `.woodpecker/*.yml`: All 5 pipeline files now use `clone.git.settings.remote: http://10.43.73.77:3000/admin/stonks-oracle.git` (Gitea ClusterIP directly, bypasses DNS)
- CoreDNS: scaled to 4 replicas, `forward . 192.168.42.1`, `dnsPolicy: None` with `nameservers: [192.168.42.1]`
- Woodpecker: `WOODPECKER_BACKEND_K8S_DNS_CONFIG` has `nameservers:[10.43.0.10]` + searches including `git-server.svc.cluster.local`

### Known Issues / TODO

1. **`stonks-oracle` namespace**: Scaled to 0 but still exists with stale data (42K extraction queue in Redis DB 0). Could be cleaned up or deleted entirely.

2. **Thesis Rewriter agent**: Was hammering vLLM from stonks-oracle namespace (5600+ calls/24h). Now stopped since namespace is scaled down. If it was also running in beta, check if recommendation service is calling vLLM for thesis rewrites excessively.

3. **`AxionML/Qwen3.5-9B-NVFP4` requests**: Something external is hitting vLLM with a model that doesn't exist (404s). Not from our pipeline — likely Open WebUI or another tool on the network configured with wrong model name. Source IP: goes through `vllm-metrics` nginx proxy (`10.42.1.155`).

4. **GitHub mirror**: `finalize.yml` mirror-github step fails (SSH key or DNS). Has `failure: ignore` so non-blocking. Needs `github_ssh_key` secret configured in Woodpecker.

5. **OpsPipeline dashboard**: Numbers now show time-filtered data. The "Document Stages" section shows counts from the selected time window (default 24h), with all-time totals as subtle subtitles. Currently beta shows: extracted=5417, low_quality=1659, parsed=15.

6. **Aggregation not generating trends on weekends**: Expected — market hours check prevents weekend trend generation. Will resume Monday.

7. **15 docs still in `parsed` status**: These are likely fresh ingests waiting for the next extraction cycle. Not stuck.

### Agent Performance (beta, last 24h as of session end)
- Document Intelligence Extractor: 33 calls, 94% success, avg 11.4s, conf 0.794
- Global Event Classifier: 81 calls, 99% success, avg 4.2s, conf 0.745
- Thesis Rewriter: 5603 calls, 100% success, avg 2.5s (from stonks-oracle before shutdown)
- Report Summarizer: 6 calls, 100% success, avg 6.9s

### Infrastructure
- k3s cluster: 4 NixOS nodes (gremlin-1 through gremlin-4)
- vLLM: `vllm-service` namespace, model `numind/NuExtract3`, 4070 Ti Super 16GB
- CoreDNS: 4 replicas, `forward . 192.168.42.1`
- Redis: DB 0 = stonks-oracle (stale), DB 1 = stonks-beta (active)
- PostgreSQL: shared instance, both namespaces use same DB server (different databases? or same? — needs verification)
- Gitea: `git-server` namespace, ClusterIP 10.43.73.77:3000, NodePort 30300
- Woodpecker: `woodpecker` namespace, kubernetes backend, 2 agents

### Key Files Modified
```
services/scheduler/app.py          — recovery thresholds + batch limit
services/validation/prediction_snapshot.py — 24h price fallback
services/aggregation/scoring.py    — normalize_impact_scores()
services/aggregation/worker.py     — normalization integration
services/trading/model_quality_gate.py — 48h threshold
services/extractor/schemas.py      — null field handling
services/extractor/main.py         — macro doc status update
frontend/src/pages/OpsPipeline.tsx  — dashboard fix
scripts/backfill_snapshot_prices.py — new script
tests/test_pbt_pipeline_health_*.py — PBT tests
tests/test_extractor_schemas.py    — updated test
infra/helm/stonks-oracle/values*.yaml — signal-engine replicas
.woodpecker/*.yml                  — ClusterIP clone fix
```