Celes Renata
a72f336ad1
Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.
70 lines
4.6 KiB
Markdown
70 lines
4.6 KiB
Markdown
# Stonks Oracle Intelligence Architecture Review
|
|
|
|
## Recommendation
|
|
|
|
Add generic OpenAI-compatible support, but implement it as a protocol/capability layer rather than a third vendor-specific branch. Keep Ollama native support. Convert the existing vLLM path into an OpenAI-compatible endpoint profile.
|
|
|
|
For the RTX 4070 Ti SUPER cluster, do not replace the current 9B model with one smaller all-purpose model. Retain the 9B model as a focused adjudicator and split routine work into CPU-first specialist stages:
|
|
|
|
1. Deterministic parsing and symbol-registry resolution.
|
|
2. GLiNER2 Large for entities, event classes, relations, and evidence spans.
|
|
3. FinBERT for company-specific financial sentiment probabilities.
|
|
4. Retrieval-based novelty and duplicate detection.
|
|
5. Calibrated confidence from observed field correctness.
|
|
6. A stock-specific tabular model trained on realized abnormal returns for impact and horizon.
|
|
7. The existing 9B Qwen-class model for ambiguous, causal, multi-company, or implied reasoning.
|
|
|
|
This preserves the current reasoning ceiling, reduces average GPU inference, improves evidence fidelity, and adds stock-specific intelligence that a general language model cannot obtain from article text alone.
|
|
|
|
## Option Review
|
|
|
|
| Option | Best use | Weakness | Production role |
|
|
|---|---|---|---|
|
|
| Current Qwen3.5-class 9B monolith | Broad zero-shot semantics and hard reasoning | Expensive per document; stochastic; self-scores confidence/novelty/impact; weak calibration | Keep as adjudicator, not universal extractor |
|
|
| Qwen3.5 4B | Smaller generalist | Lower reasoning ceiling with same architectural weaknesses | Benchmark only; not preferred |
|
|
| NuExtract 1.5 3.8B | Literal schema filling | Limited implicit market reasoning; adds another generative runtime | Optional benchmark/fallback |
|
|
| NuExtract 1.5 Smol 1.7B | Compact long-form extraction | Still autoregressive and not a sentiment/impact model | Optional CPU/on-demand filing stage |
|
|
| NuExtract Tiny 0.5B | Very small extraction experiments | Accuracy ceiling too low for authoritative trading inputs without task tuning | Research/fine-tuning baseline |
|
|
| GLiNER2 Large 340M | CPU-first entities, classes, relations, spans | Needs calibration and task-specific tuning for best results | Primary fast-path specialist |
|
|
| FinBERT | Financial positive/negative/neutral probabilities | Not an extractor or reasoner | Per-company evidence sentiment |
|
|
| Hybrid specialist + 9B | Routine precision plus retained hard-case intelligence | More engineering and observability work | Recommended architecture |
|
|
| Hybrid + historical impact model | Text intelligence plus actual market-response learning | Requires leakage-safe dataset and monitoring | Best end-state |
|
|
|
|
## Highest-Priority Existing Problems
|
|
|
|
1. `services/extractor/vllm_client.py` ignores the supplied schema and requests only a generic JSON object.
|
|
2. The vLLM default extraction temperature is `0.7`.
|
|
3. Unknown provider values silently route to Ollama.
|
|
4. Documents are truncated to 8,000 characters.
|
|
5. The model is asked to invent authoritative novelty, confidence, impact, and horizon values.
|
|
6. Those self-scores directly affect aggregation weighting.
|
|
7. Provider attribution is hardcoded to Ollama.
|
|
8. The extractor processes one job at a time at the application layer.
|
|
9. Endpoint/model defaults conflict across code, migrations, Helm, and the vLLM deployment.
|
|
10. A tracked Helm override contains plaintext production-like credentials and requires immediate rotation.
|
|
|
|
## Expected Performance Shape
|
|
|
|
The following are design targets to validate, not promises:
|
|
|
|
- 60-80% of representative documents accepted through the CPU fast path after calibration.
|
|
- 2x or greater reduction in GPU-seconds per accepted document.
|
|
- Peak GPU memory near the current 9B deployment because no second generative model is permanently GPU-resident.
|
|
- p50 latency substantially lower for routine documents.
|
|
- p95 latency near the current model path for adjudicated documents.
|
|
- Better exact-field and evidence accuracy from deterministic/specialist stages.
|
|
- Same broad semantic ceiling because the 9B model remains available.
|
|
- Better impact/horizon calibration once the historical outcome model is approved.
|
|
|
|
## Immediate Next Decision
|
|
|
|
The first implementation milestone should not be GLiNER integration. It should be:
|
|
|
|
1. Rotate exposed credentials.
|
|
2. Establish the real runtime model/configuration.
|
|
3. Fix strict JSON Schema output and temperature on the current 9B endpoint.
|
|
4. Build the gold corpus and replay harness.
|
|
5. Then implement the gateway and specialist shadow path.
|
|
|
|
That order creates a fair baseline and prevents the project from attributing simple request fixes to the new architecture.
|